Effects of Comprehensive Background-Check Policies on Firearm Fatalities in 4 States

Abstract

Despite promising results from individual-level studies, state-level studies of the effectiveness of comprehensive background-check (CBC) policies in reducing firearm fatalities have yielded null results in multiple states. These prior studies focused on CBC laws adopted in the 1990s, when record keeping was far less complete. We estimated the effect of the implementation of CBC policies on state-level firearm homicide and suicide rates in states implementing CBC policies from 2013 to 2015 (Colorado, Delaware, Oregon, and Washington). We compared age-adjusted firearm homicide and suicide rates, measured annually from 15 years prior to policy implementation until 2019, in each treated state to rates in control groups constructed using the synthetic control group method. Differences in firearm homicide rates for Colorado, Oregon, and Washington post treatment were all small (0.09 to 0.18 per 100,000 residents per year) and not well distinguished from natural variation. Oregon had on average 0.80 per 100,000 fewer firearm suicides per year than did synthetic Oregon post treatment. However, these results were inconsistent across modeling approaches and not well distinguished from natural variation. Our models produced poor fit for Delaware. Coupled with previous null results from Indiana, California, Maryland, Pennsylvania, and Tennessee, the present results suggest that extending background check requirements to private transfers alone and implementing these policies as is currently done is not sufficient to achieve significant state-level reductions in firearm fatalities.

Abbreviations

CBC

comprehensive background check

RMSPE

root mean square prediction error

Firearm morbidity and mortality are major public health and safety concerns in the United States. In 2020, there were more than a quarter million violent victimizations with a firearm (1), and 44,286 people were killed by firearm violence, more than half of whom died by suicide (2). The objective of the present study was to estimate the effect of the implementation of comprehensive background-check (CBC) laws on state-level firearm homicide and suicide rates in 4 states enacting CBC laws from 2013 through 2015: Colorado (2013), Delaware (2013), Oregon (2015), and Washington (2014).

Under federal law, individuals who want to purchase handguns or other firearms from a licensed retailer must pass a background check verifying that they do not have any prohibiting conditions (e.g., felony convictions); however, sales by unlicensed sellers (i.e., private sales or transfers between individuals) are exempted from this requirement. Data from national surveys suggest that roughly one-third of firearm transfers are between private parties (3) and that approximately 50% of private transfers occur without a background check (4). This estimate is much lower (26%) in states with CBC laws (4).

As of December 2021, 21 states and the District of Columbia have extended the background check requirement to the private transfer of handguns. Eleven states have so-called CBC laws that extend background-check requirements, at least for handguns, to sales by private sellers, using a mechanism similar to that required by federal law. Under CBC laws, potential purchasers apply through a licensed firearms dealer who submits the application to law enforcement agencies to perform the background check. Another 10 states and the District of Columbia implemented a private sale background check by requiring prospective purchasers to first obtain a permit for purchase, which also involves a background check. Of these 10 states, 4 states and the District of Columbia require prospective purchasers to acquire a permit and undergo a background check at the point of sale.

Research on background checks suggests that denying firearm purchase to individuals who are legally prohibited from gun ownership leads to reductions in the risk of violence among those denied individuals (5, 6). One study found that individuals prohibited from purchasing a firearm because of a misdemeanor violent offense were 22% less likely to be arrested for a subsequent violent offense when compared with individuals with a similar criminal history who were not denied purchase (6). A study focused on mental health-related firearm prohibitions similarly found a protective association when the relevant records were stored in the appropriate background-check databases (7).

Another body of research has sought to estimate the effects of CBC laws (applicable to at least all handgun sales) on state-level rates of firearm homicide and suicide. Longitudinal studies that seek to isolate the independent associations between the enactment or repeal of CBC laws and changes in firearm mortality rates have not identified such associations (8–10).

These studies are distinct from the previous studies in that they estimate effects of a policy change (rather than effects of firearm purchase denial) on state-level mortality rates (rather than risk among persons meeting prohibiting criteria).

Some states enhance CBC policies by requiring permits or licenses to purchase firearms. Purchaser licensing laws typically require prospective firearm or handgun purchasers to apply directly with law enforcement agencies and often require fingerprinting and allow more time to complete background checks, both of which improve the rigor of background checks. States implementing (or repealing) permitting requirements have seen decreases (and increases, respectively), in rates of firearm homicide and firearm suicide relative to control groups (11–14). Policy effectiveness may depend on specific elements of design, implementation, and enforcement.

The present study focused on CBC policies without permitting requirements. Previous studies of the effectiveness of CBC policies were conducted in states with nascent versions of modern CBC laws. Indiana and Tennessee repealed their CBC laws in 1998, and California implemented its law in 1994. Importantly, early iterations of CBC policy were not strictly enforced and relied on databases with incomplete records (15). For example, in the 1990s, missing criminal history and mental health records in the background-check systems allowed people to purchase firearms even when they met prohibiting criteria (15). Recordkeeping has since improved (e.g., a greater proportion of records are available in record-keeping systems, fewer records are stored solely on paper, and dashboards allow for more and faster data sharing across agencies) (16), and it is important to understand the effectiveness of CBC laws under these improved conditions.

METHODS

We used a quasi-experimental study design comparing each treated state with a control group constructed using the synthetic control group method (17).

Sample

Treated states included all those having implemented CBC policies during 2013–2015: Colorado (2013), Delaware (2013), Oregon (2015), and Washington (2014). States implementing CBC laws more recently than 2015 were excluded due to the limited availability of outcome data following policy implementation. Vermont implemented its CBC law in 2017, Nevada began enforcing its law in 2020, and New Mexico’s took effect in July of 2019.

Donor pool states (those eligible to serve as controls) were all states without a CBC or permit-to-purchase law for the entire study period (n = 28). The start of the study period was set at 15 years prior to the enactment date for each state and extends through 2019.

Data

The exposure was state adoption of a CBC law. Dates on which the laws took effect were determined based on state statute. We collected state statutes, legislation, and regulatory documents using standard search terms in major legal databases: Westlaw (Thompson Reuters Corporation, Toronto, Ontario), HeinOnline (Getzville, New York), and Nexis Uni (LexisNexis Corporation, New York, New York). In addition, we examined state-specific websites. A state was determined to have a CBC law if it had a policy that required background checks for at least all handgun sales, including both private purchases and purchases from licensed dealers. States that required such background checks as part of a licensing system were excluded. We compared the states we identified with existing gun law collections as a check on the validity of our findings (https://www.statefirearmlaws.org/ and https://giffords.org/). Implementation dates were defined as the date that a new law went into effect, not the date a new law was signed or enacted. The first year of the post-treatment period for each state began in the year of implementation or in the subsequent year if the implementation date was after July 1st. The full data set of relevant firearm laws is available on OpenICPSR (18).

Outcomes included firearm and nonfirearm homicide and suicide rates by year and state. We did not expect CBC law implementation to affect rates of nonfirearm homicide or suicide, and thus these outcomes were conceptualized as negative controls. These data were obtained from the National Vital Statistics System, which relies on death certificate data and constitutes the most complete data on deaths in the United States (19). We calculated age-adjusted rates using the 2000 US population as the standard population. Population estimates are available from the Centers for Disease Control and Prevention’s Wide-Ranging Online Data for Epidemiologic Research (WONDER) database (20).

Models for firearm and nonfirearm homicide included state demographic composition (% male, % age 18–29, % Black or African American, % Hispanic or Latino) (20), socioeconomic indicators (% of households earning below the poverty line (21), % of civilian labor force that is unemployed (22), % of population over age 25 years with a high-school diploma (23–34), Gini coefficient measuring income inequality (35, 36), per capita personal income (37)), a proxy for firearm ownership (38, 39) (% of suicides completed with a firearm), % of population living in an urban area (40), per capita ethanol consumption (41), sworn police officers per 100,000 residents (40), violent crimes per 100,000 residents (40), and imprisonment rate (per 100,000) (42–45). Models for firearm and nonfirearm suicide additionally included % of population over age 18 years that are veterans (46, 47) and the number of people practicing a religion (per 1,000) according to religious organizations (48), and excluded incarceration and violent crime rates. Variables were chosen based on their known or hypothesized associations with the outcomes and their use in previous studies (8–10, 13, 49–51). Pre-intervention measures of the outcomes were also included in our models.

Three variables (Gini index, % of population that are veterans, and the number of people who adhere to a religion per 1,000 residents) were available in fewer than 20% of the study period years. These variables were excluded from the main analyses but were included in supplementary analyses. The percentage of the population with a high-school degree or higher was missing in 2008 for all states (n = 32, 4.5%), the percentage of the population that is Latino or Hispanic was missing in 5 state-years (0.7%), the percentage of the population between the ages of 18 and 29 years and the count of sworn officers per 100,000 residents were missing in 2 state-years (0.3%), and incarceration rates were missing in 1 state-year (0.1%). These values were imputed using a multivariate linear imputation strategy and the variables were retained in final analyses. No observations for any of the control variables met our definition of an outlier (>5 standard deviation from the mean), and all pairwise correlation coefficients were under 0.8, with the exception of individual income and year (0.84).

Analytical approach

We used the synthetic control method to generate an estimate of the counterfactual trend in the outcome following policy implementation (17, 52). This method has been used widely to estimate effects of policy changes and specifically, changes in firearm policy (8, 9, 11, 13). The estimate of the counterfactual trend is based on a weighted average of the outcome in control states following policy change, where the weights are calculated to minimize the difference between the weighted average in the synthetic state and the true state, prior to policy change. We used data from the 15 years prior to the policy change to generate these weights. Some states in the donor pool received a weight of zero and did not contribute to the synthetic state. We performed separate synthetic control analyses for each of the 4 CBC states and for each of the 4 outcomes. We compared the mean of the outcome over the years following the policy change in each observed treated state to the same mean from each state's synthetic control using absolute and relative measures.

Placebo tests were used as a point of comparison for the magnitude of the difference in each outcome between each treated state and its synthetic control. These placebo test results were estimated by comparing each state in the donor pool with its own synthetic control, excluding all treated states from these comparisons. The percentage of placebo tests for which the difference between the observed values and the synthetic control was greater than that of the CBC states served as a pseudo-P value. States for which the synthetic control was poorly fit (>5 times the root mean square prediction error (RMSPE) of the treated state analysis) were dropped from the calculation of this percentage following the example of Abadie et al. (17)

We additionally used the augmented synthetic control method with ridge regression proposed by Ben-Michael et al. (53) to estimate differences after policy implementation. Augmented synthetic control allows for negative donor pool weights in order to more flexibly fit pre-intervention trends and allows for standard hypothesis testing. We manually converted P values from this output to confidence intervals (54).

Sensitivity analyses

The weights assigned to donor states can be sensitive to the number of pre-intervention years, the inclusion of specific donor states, the set of predictor variables, and the lagged values of the outcome included (e.g., average lag, different years of lags, number of lags). Additional analyses vary these parameters to assess the robustness of our findings to changes in the estimation strategy.

Changes in state stand-your-ground laws (which allow persons to use deadly force to defend themselves or others in public even if they could safely retreat (55)) and shall-issue concealed-carry license laws (requiring states to issue concealed-carry permits to all applicants who are legally permitted to own a firearm (56)) have been shown to be associated with changes in the rates of gun violence at the state level. Sensitivity analyses excluded states experiencing changes in these laws in the 2 years prior to policy implementation through 2018. These include the following, with policy and implementation year in parentheses: Idaho (permitless carry, 2016), Kansas (permitless carry, 2015), Maine (permitless carry, 2015), Mississippi (permitless carry, 2016), North Dakota (permitless carry, 2017), West Virginia (permitless carry, 2016), Wisconsin (shall issue, 2011), Wyoming (permitless carry, 2011; stand your ground, 2018).

Secondary analyses

We conducted similar analyses using a more traditional regression approach. To select the model, we performed simulations following the recommendation of a RAND Corporation report to identify the regression model that minimized false positives and false negatives for our research question (50). We tested 4 models. Each was fitted with Poisson and negative binomial distributions. These analyses included all the same control variables listed above except for gun availability and the violent crime rate, which may be influenced by the enactment of CBC policies. We also controlled for the presence of CBC and other firearm policies (permit-to-purchase, stand your ground, and concealed carry). Also following the example provided by RAND, we allowed the policy change to phase in over 3 years (50). The change-coded model with state-specific lags and fit using negative binomial regression performed the best across all 4 of our outcomes and was applied for this secondary analysis.

Other secondary analyses restricted the outcome to high-risk subsets of the population. These include: 1) firearm and nonfirearm homicide among young men ages 20–39 years and 2) firearm and nonfirearm suicide among men aged ≥60 years.

RESULTS

Descriptive statistics

Average firearm homicide rates in the 4 treated states over the study period ranged from 1.7 per 100,000 per year in Oregon to 4.1 per 100,000 in Delaware, and average firearm suicide rates ranged from 5.1 in Delaware to 9.0 in Colorado (Figure 1).

Figure 1.

Firearm homicide (A) and firearm suicide (B) rates (per 100,000 per year) in each treated state (Colorado, Delaware, Oregon, and Washington) and the average of control states (28 US states without comprehensive background-check laws), 1998–2019.

Synthetic control analysis results

The pre-intervention fit between synthetic Delaware and the observed values in Delaware was quite poor in all iterations of the synthetic control analysis, including augmented synthetic control (Web Figures 1–16, available at https://doi.org/10.1093/aje/kwac222), for both firearm homicide and suicide rates (RMSPEs = 0.978 and 0.728, respectively). As a result, results for Delaware are unreliable and we omit them from further discussion. This conclusion is further supported by post hoc tests of trend for the difference in the rates of firearm homicide and suicide between Delaware and synthetic Delaware in the pretreatment period. The trend in the difference in rates of firearm homicide was 0.13 (P value = 0.031) and for firearm suicide it was −0.12 (P value = 0.001), providing evidence that the parallel trends assumption was not met for these models. Results for the other states under consideration provided coefficients of <0.02 and P values that ranged from 0.338 to 0.953.

The weights assigned to donor pool states for each treated state and the comparisons of pre-intervention variables for each treated state and its synthetic control are presented in Web Tables 1–16.

Firearm homicide

With the exception of Delaware, the RMSPEs for firearm homicide rates were small, indicating good fit, and ranged from 0.136 in Washington to 0.274 in Colorado (Table 1). The magnitude of the differences in observed and forecasted firearm homicide rates following the implementation of CBC policies in Colorado, Oregon, and Washington were all small and well within what would be expected given natural variation (Table 1; Figure 2; Web Figures 17–22). Sensitivity analyses in which we varied the parameters of the synthetic control approach showed similar results (Web Figures 23–37), as did the results from the negative binomial regressions (Web Table 17) and augmented synthetic control (Web Figures 38–40). Results for men aged 20–39 years were larger in magnitude but still within the bounds of expectation given estimated natural variation (Web Table 18).

Table 1.

Model Fit and Average Differences Between Treated States and Their Synthetic Controls During the Postintervention Period in Colorado (2013–2019 ), Delaware (2014–2019), and Oregon and Washington (2015–2019)

Fatality Type and State	RMSPE	Difference	Relative Rate	% Placebo Tests > Difference	No. of Placebo Tests > Difference
Firearm suicide
Colorado	0.645	0.20	1.02	82	23 of 28
Delaware	0.728	−0.82	0.87	21	6 of 28
Oregon	0.331	−0.80	0.92	27	7 of 26
Washington	0.325	−0.13	0.98	81	21 of 26
Firearm homicide
Colorado	0.274	0.18	1.07	70	19 of 27
Delaware	0.978	0.42	1.08	39	11 of 28
Oregon	0.211	0.09	1.05	89	24 of 27
Washington	0.136	0.17	1.08	78	18 of 23

Abbreviation: RMSPE, root mean square prediction error.

Figure 2 .

Continues

Firearm suicide

The synthetic controls for firearm suicide rates in Washington and Oregon fit moderately well (RMSPEs = 0.325 and 0.331, respectively); the fit for Colorado was fairly poor (RMSPE = 0.645) (Table 1; Figure 2). Differences in Colorado and Washington were small and within the range of differences produced by the placebo tests (Web Figures 41–46). In Colorado, results from some sensitivity analyses were suggestive of an increase in the rate of firearm suicides, although differences were within the range of natural variation as estimated by placebo tests (Web Figures 47–51).

During the postintervention period, Oregon had on average 0.80 per 100,000 fewer firearm suicides per year than did synthetic Oregon (equivalent to 8% or approximately 34 fewer suicide deaths per year) (Table 1). This difference was larger in magnitude than the differences observed in Colorado and Washington but still within the range of expectation suggested by the placebo tests. Results from sensitivity analyses using synthetic control showed similar patterns (Web Figures 52–56; sensitivity analyses for Washington are available in Web Figures 57–61). The differences from the augmented synthetic control model for Oregon (0.55 per 100,000 fewer firearm suicides per year, 95% confidence interval: = −3.380, 2.280) and using negative binomial regression (coefficient = 0.03, 95% confidence interval: = −0.204, 0.253) were smaller in magnitude and could not be distinguished from random variation in the outcome measure (Web Figures 62–63, Web Table 17). Finally, in the models including only men aged 60 years or older, RMSPEs were quite large, indicating poor fit (Web Table 18).

DISCUSSION

Our results were null, with a few inconsistencies suggestive of a small increase in firearm suicide in Colorado and a decrease in Oregon, although these differences were similar in magnitude to differences observed in the placebo tests and sensitive to model specification. Additionally, neither finding was replicated when focusing on rates among men aged 60 years or older, the group at highest risk. We also did not observe changes greater than expected in firearm homicide rates. Given the variability in rates over time, the number of tests performed, and the lack of robustness to modeling approach, any inconsistent results are unlikely to reflect differences in effect that depend on state, outcome, or age.

Previous findings on the state-level association between CBC policy implementation (8, 10) or repeal (9) and firearm homicide and suicide rates were similarly null. Given the consistent evidence in individual-level studies of a protective association between background checks and firearm violence (5–7), and in state-level studies of the effectiveness of permit-to-purchase policies (10–13), it is important to consider the statistical and practical reasons that may underlie these state-level findings.

Figure 2.

Firearm homicide and suicide rates per 100,000 per year in each treated state and its synthetic control for Colorado, Delaware, Oregon, and Washington, 1998–2019. The vertical lines indicate the timing of policy implementation.

In Colorado, Delaware, Oregon, and Washington, robust data systems for collecting and maintaining records (16), one of the motivating reasons for the present study, reduce the probability that a prohibited person passes a background check due to missing records or delays. All have fully automated criminal history records (i.e., they do not rely on paper records), all enter protective orders into the National Crime Information Center (NCIC) protection order file within a week of issuance, and, with the exception of Colorado, more than 80% of felony arrests have corresponding final case dispositions (16). However, background checks can only potentially prevent shootings by individuals prohibited from legal firearm purchase. Data from surveys of inmates of state prisons who were incarcerated for committing violent crimes with firearms reveal that a significant share of these offenders, in some states the majority, were not prohibited from possessing the firearm that they used in their offense (57).

Second, people circumvent the background-check system through straw purchases or simply by purchasing from sellers who do not conduct background checks even when checks are required. Previous research on National Instant Criminal Background Check System (NICS) background checks suggests that the uptake of background checks among private sellers and buyers is variable. In Delaware and Oregon, NICS checks increased consistently over time, relative to controls, following the implementation of CBC policies (49, 58). For Colorado and Washington, however, there either was no observable increase in background checks (49) or the increase was gradual and could not be distinguished from the null (58). Yet even if background checks increase with CBC laws, the protective effects of those checks will be muted if applied only in the case of low-risk transfers.

Having a firearm in the home is associated with an increased risk of suicide (59), as is acquiring a firearm through legal means (60). Research on firearm suicide risk associated with illegal private party purchases, however, is lacking. Most prohibiting conditions for firearm purchase are based on prior acts of violence against others or risks for committing such acts. While a history of violence and substance abuse are also risk factors for suicide, it is likely that a large share of those at risk for committing suicide are not prohibited from legal firearm possession. CBC laws alone may be insufficient to prevent a significant share of those at risk for suicide from accessing a firearm. It is also possible that background checks prevent so few firearm suicides that it would be impossible to detect differences at the state level. Finally, if background checks are not completed even when they are legally required—or people circumvent the law by, for example, having someone purchase a firearm on their behalf—even high-risk prohibited individuals can obtain firearms and use them to harm themselves. However, laws that are more likely to deter impulsive purchases (for example, acquisitions when someone is contemplating suicide), such as waiting period and permit-to-purchase laws, are associated with reductions in firearm suicides without significant method substitution (10, 11, 61).

This study uses a strong set of controls, multiple comparison years, sophisticated and multiple modeling approaches, and reliable measures of firearm homicide and suicide. However, the sample size is relatively small, with 1 treated state at a time and 28 states available as controls. This is particularly problematic when other factors with a plausible association with the outcomes co-occur with treatment. For example, Oregon legalized recreational marijuana use in 2015, the same year as the policy change presently under study. As such, we cannot separate the effects of one policy change from the other. Other violations of the common shock assumption are also possible if unobserved time-varying confounders affect the treated and control groups differently.

The outcome, fatal firearm violence, is also relatively rare, making it harder to detect differences if they do in fact exist. Future research should consider nonfatal violence as an outcome as the associations may differ.

Finally, the synthetic control method is limited by the use of placebo tests to generate the measure of statistical inference. Placebo tests require that the variance of effect sizes across the actual and placebo tests does not depend on variables that would differentiate the association of interest from the associations produced through the placebo tests. In fact, many variables may be associated with the standard error of the synthetic control estimate (e.g., population size, frequency of outcome) and differ across treatment and placebo effect estimates. We therefore allowed P values generated from augmented synthetic control and traditional regression models to contribute to our interpretations of statistical inference.

In conclusion, previous research supports the effectiveness of background checks among people meeting denial criteria (5–7), yet many state-level studies have failed to identify an effect on fatal outcomes (8–10). The present study is, to our knowledge, novel in its focus on states with fairly robust record keeping systems, yet results largely remain null. As such, the predictive ability of denial criteria, the enforcement of CBC policies, and other methods for promoting compliance with CBC laws are ripe targets for future research into the effectiveness of CBC policies.