Gun Shows and Gun Violence: Fatally Flawed Study Yields Misleading Results

Abstract

A widely publicized but unpublished study of the relationship between gun shows and gun violence is being cited in debates about the regulation of gun shows and gun commerce. We believe the study is fatally flawed.

A working paper entitled “The Effect of Gun Shows on Gun-Related Deaths: Evidence from California and Texas” outlined this study, which found no association between gun shows and gun-related deaths. We believe the study reflects a limited understanding of gun shows and gun markets and is not statistically powered to detect even an implausibly large effect of gun shows on gun violence. In addition, the research contains serious ascertainment and classification errors, produces results that are sensitive to minor specification changes in key variables and in some cases have no face validity, and is contradicted by 1 of its own authors’ prior research. The study should not be used as evidence in formulating gun policy.

IN EARLY OCTOBER 2008, THE National Bureau of Economic Research posted on its Web site a working paper by Duggan et al. titled “The Effect of Gun Shows on Gun-Related Deaths: Evidence from California and Texas.”¹ A University of Michigan press release announced the study with the headline “Gun shows do not increase homicides or suicides” and began as follows:

A new study finds no evidence that gun shows lead to substantial increases in either gun-related homicides or suicides. The…study also shows that tighter regulation of gun shows does not appear to reduce the number of firearms-related deaths.²

Not surprisingly, the paper received nationwide publicity. Just 6 weeks after its release, it was the topic of a relentlessly favorable cover article in America's 1^St Freedom, the monthly official journal of the National Rifle Association.³

To our knowledge, the working paper has not been published in a peer-reviewed publication. Anecdotal reports from policy advocates and policymakers suggest that it has nonetheless become influential in the continuing debate about how best to regulate gun shows and gun commerce to prevent violence. We believe that the study discussed in the working paper contains serious errors in design and execution that fatally compromise its findings. Correspondence with the authors^4,5 did not lessen our concerns and, because the paper remains in circulation, we seek to share those concerns with the research and policy communities. For brevity's sake we confine our comments largely to the study's findings for homicide, although we have similar concerns about the study's findings for suicide.

LIMITS TO THE STUDY DESIGN

Large gun shows can usefully be thought of as the big box retailers of gun commerce.^6,7 Dozens to hundreds of gun sellers, both licensed retailers and private parties, are present and compete for the business of thousands of customers. Surveys suggest that from 4% to 9% of gun sales nationwide occur at gun shows.^8,9 Most sales at gun shows are made by licensed retailers, and most private-party gun sales are made elsewhere.^6,7 Federal law makes no distinction between sales occurring at gun shows and other sales; there is no such thing as a gun show loophole. Sales by licensed retailers require a background check, substantial paperwork, and a permanent record, but sales by private parties can be completely anonymous and undocumented.¹⁰ Seventeen states require background checks for private-party handgun sales at gun shows, and 10 do so for gun show sales of rifles and shotguns.¹⁰ (Only 6 states require background checks for all private-party gun sales.¹⁰)

The relationship between gun shows and gun-related crime is complex. Although they are a source of the guns used in crime, gun shows are just 1 of the venues that supply firearms for criminal use on a more-or-less continuing basis. Surveys of persons incarcerated for gun crimes suggest that less than 2% of them acquired their weapons at gun shows.^11,12 Gun shows are more frequently an intermediate source of crime guns, supplying organized gun trafficking operations and surrogate or “straw” purchasers who in turn supply criminal end users.^6,7,13,14 A review of federal investigations initiated between July 1996 and December 1998 of illegal gun trafficking found that nearly 26 000 trafficked guns—30% of all guns linked to the investigations—had been sold at gun shows.¹⁵

Duggan et al. hypothesized that gun shows have short-term and local effects on homicide (and suicide). Their data are for California and Texas from 1994 to 2004. To test their hypothesis, they determined whether there was any association between the number of deaths in a given location (specified as a zip code) in a given week and the number of gun shows in or near that location in the preceding 4 weeks. Their unit of analysis is the zip code × week, and their models variably examined zip codes for areas in which gun shows occur and codes for areas that were within 5, 10, and 25 miles of gun shows. Duggan et al. acknowledged these time and space restrictions as a limitation,¹ but in our judgment the restrictions lead to fundamental questions about the study's design and the validity of its findings.

Many studies have established that gun crimes rarely involve guns purchased within a few weeks of the crime's occurrence.^16–18 Those studies relied on data from gun tracing, which is

[T]he systematic process of tracking a recovered crime gun's history from its source (manufacturer/importer) through the chain of distribution (wholesaler/retailer) to the individual who first purchases the firearm.^16(pA4)

In preparing this critique we have analyzed tracing data compiled by the Bureau of Alcohol, Tobacco, Firearms and Explosives for guns recovered in California and Texas from 2003 to 2006. Only 1.3% of traced crime guns from those states were recovered by law enforcement agencies within 4 weeks of their first retail sale.

Similarly, Duggan et al. restrict their analysis to events occurring within at most 25 miles of a gun show, which ignores well-established findings about the geography of illegal gun markets. As a general rule, as many as one-third of crime guns recovered in most cities are purchased out of state, and another third are purchased in the same state but outside the immediate vicinity.¹⁶ Our analysis of data for traced guns recovered in California and Texas from 2003 to 2006 finds that 49.5% were recovered within 25 miles of their point of initial sale. Additional data from that study, which is still in progress, suggest that guns purchased at gun shows travel an even greater distance from their point of sale to their place of use in crime than do guns purchased in other venues. (G. P., A. B., and G. W., unpublished manuscript, 2010).

The combined impact of the temporal and geographic constraints imposed by Duggan et al. is demonstrated in Table 1. From 2003 to 2006, just 1.0% of guns involved in crimes in Texas and California were recovered within 28 days and 25 miles of their entry into the market.

TABLE 1.

Time and Distance From First Sale by a Licensed Retailer to Recovery by a Law Enforcement Agency for Guns in California and Texas: 2003–2006

Time from Gun Sale to Recovery by Police	Miles from Purchase to Recovery
	0–25, No. (%)	≥26, No. (%)	Total, No. (%)
1–28 d	680 (1.0)	284 (0.4)	964 (1.4)
29–90 d	1009 (1.4)	442 (0.6)	1451 (2.1)
91–365 d	2710 (3.9)	1699 (2.4)	4409 (6.3)
≥366 d	30 260 (43.2)	32 894 (47.0)	63 154 (90.2)
Total	34 659 (49.5)	35 319 (50.5)	69 978 (100.0)

Note. Percentages are of all guns in the table. Data were missing for another 79 122 guns. For a gun to be added to the table, dates and places for both sale and recovery were required. Data were missing for a variety of reasons: a retailer may not be able to provide a date of sale, for example, or the serial number on the gun may have been obliterated, making tracing impossible. Some traces for handguns recovered in California may have been extended beyond the first retail sale, as the state maintains a computerized archive of handgun purchases that is used for crime gun tracing. With few exceptions, traced guns have been used in a crime or recovered under circumstances that suggest criminal use.

Source. G. Pierce, A. Braga, and G. Wintemute, unpublished manuscript, 2010.

It is important to note as a limitation to our analysis that standard gun traces terminate with the first retail sale of a gun, and both new and used guns are sold at gun shows. But when gun traces are extended to the most recent recorded sale—as is possible for handguns in California, for which a centralized archive of sales data exists and where private-party sales must be routed through licensed retailers—overall results are similar to those obtained using standard gun traces.¹⁸

It is entirely possible that gun shows have long-term and far reaching effects on homicide (and suicide), without exhibiting the short-term, local effects hypothesized by Duggan et al. In most urban areas gun shows occur less than once a month.⁶ Potential offenders who want a gun immediately are unlikely to wait for a gun show. As discussed, gun shows may be a more likely choice for criminal gun trafficking operations, which often move guns long distances across state lines or our borders with Mexico and Canada.^6,13

LACK OF STATISTICAL POWER

The time and distance constraints imposed by Duggan et al. create an insurmountable statistical power problem. Given those constraints and the data in Table 1, it is reasonable to infer that their analysis misses the great majority—as much as 99%—of the expected effect of gun shows on homicide. The following highly favorable power calculation finds that the analyses by Duggan et al. have less than 10% power for finding even a very large effect of gun shows on homicide.

For the power calculation, let us err in favor of the original study and assume that (1) 5% of homicides committed with guns purchased at gun shows occur within 4 weeks and 25 miles of those shows, and (2) the total effect of gun shows is to increase gun homicide by an implausibly large 30%, regardless of time and place. This means the study is attempting to identify a 1.5% increase in homicides (5% × 30%). Let us also assume that the control group size is 900 000 (the maximum control group size for California), and the size of the intervention group (gun show counties) is 20 000.

Using findings from Duggan et al.¹ of a mean gun homicide rate per county of 0.035 deaths per 100 000 persons per week with a standard deviation of 0.203, then for a 2-sample test, 1-tailed, with an α of 5%, the power is 10%.

In other words, Duggan et al. had only a 10% chance of detecting even an implausibly large effect of gun shows on homicide. Given this result, the fact that the they did not find an effect of gun shows on homicide cannot be taken to mean that no such effect exists. As we have suggested, that effect may simply be exerted over a longer time period and broader geographic area.

DATA DEFICIENCIES

Duggan et al. used 1 publication, the Gun & Knife Show Calendar, to identify gun shows. But prior research^6,14 shows that no single source is adequate for this purpose. Another widely available listing is in the Big Show Journal. To determine the magnitude of misclassification that might exist in data obtained from a single source, we compiled data from both publications for 2007 and found that 298 gun shows were held in California or Texas that year. Only 236 (79%) of these were listed in the Gun & Knife Show Calendar. This suggests that Duggan et al. failed to identify roughly 20% of the gun shows occurring in those states during their study period—some 680 events altogether. Based on the listings for 2007, some of these missed shows would have been large events with many gun sales.

If, as Duggan et al. hypothesize, gun shows are associated with a short-term, local increase in homicide, then failing to identify 20% of gun shows—and as a result wrongly assigning homicide data for the weeks following those missed shows to the no-gun-show category—introduces a bias toward the null (a finding of no effect) in their analysis.

There is significant heterogeneity among gun shows,^6,14 but Duggan et al. treat them equally. Smaller shows can involve fewer than a dozen gun sellers, all of them small-scale operations and most of them selling only used guns. Such events may have only a few dozen attendees on site at any time. Some shows are sponsored by gun collector organizations and display primarily antique weapons with little, if any, utility for criminal purposes. Large general-purpose gun shows involve hundreds of gun vendors, and individual vendors may have more than 1000 new guns on display.⁶ Any impact of these large shows might easily be masked by the limited effect of the more numerous smaller events.

CONFLICTING RESULTS AND PRIOR RESEARCH

An earlier version of the study by Duggan et al. yielded different results.¹⁹ In that study, the relevant portion of which was limited to gun shows in California, census place and not zip code was used as the geographic unit of analysis and the authors argued specifically that zip code was a less appropriate choice. In the earlier version, there was also a statistically significant overall increase in gun deaths in the 4 weeks following gun shows. When deaths were disaggregated by type, the effects remained significant for suicide and unintentional death. Nothing is said in the new version of the study about the prior and very different findings or in rebuttal to their own prior arguments against using zip code as a geographic unit of analysis. We do not take a position on whether zip code or census place is the more appropriate measure, as our concern is with the larger issue of geographic restriction. The unexplained reversal and the lack of robustness of the study findings are independent sources of concern about the validity of those findings.

An entire body of literature, reviewed elsewhere,^20,21 has documented a positive relationship between gun ownership and both gun homicide and gun suicide. One of these studies, though it is not often cited in this context, was conducted by Duggan.²² In that study, titled “More Guns, More Crime,” Duggan used sales of Guns & Ammo magazine as his proxy measure for gun ownership. He demonstrated the validity of his proxy measure in part by demonstrating that state-level magazine sales were very closely linked to a state-level count of gun shows. The regression coefficient was 0.995; a 10% change in either measure was associated with a 10% change in the other. As a practical matter, the study would have found a nearly identical relationship between gun ownership and gun crime had the number of gun shows, not the number of magazines sold, been used as a proxy for gun ownership. Such a study might have been titled, “More Gun Shows, More Crime.”

CONCLUSIONS

Taking all these points together, we believe the study by Duggan et al. is fatally flawed. The specifics of the hypothesis tested in the study essentially precluded the possibility of a positive finding, and the statistical power problem in itself is sufficient to render the study's results essentially incapable of interpretation.²³ Beyond this, the data used for the analysis contain important ascertainment and classification errors that introduce biases toward the null, and the results are not robust.

Given these difficulties, it is perhaps not surprising that some of the study's findings have no face validity. For example, in some models Duggan et al. find significant changes in gun-related deaths in the week beginning on the first day of the gun show for shows occurring in California. But California has a 10-day waiting period for all gun sales, including those at gun shows; transfers of possession cannot legally occur during that first week.

Finally, we note that Duggan et al. argued based on their findings that regulating gun shows has no effect. But California's regulations on gun shows produce observable effects that are likely to benefit public safety, including decreases in anonymous, undocumented private-party gun sales and illegal surrogate or “straw” purchases.¹⁴ It has also been shown that regulating private party gun sales generally, not just at gun shows, is strongly associated with a decrease in intrastate gun trafficking, even when gun ownership rates and other gun regulations are taken into account.²⁴