The National Violent Death Reporting System and Police-Involved Firearm Deaths

The Conner et al. (p. 578) article in this issue of AJPH has two aspects. First, it uses data from the National Violent Death Reporting System (NVDRS) to examine the frequency of fatal shootings by the police. Second, it evaluates the quality of NVDRS data for addressing this difficult to study but nationally important issue. Both aspects are significant contributions to the literature.

Research conducted over several decades has documented the underreporting of legal intervention homicides by the police.^1–3 The limitations of the available data became a matter of national concern in 2014, after Michael Brown died of an officer-involved shooting in Ferguson, Missouri. In reacting to Brown’s death, then-director of the Federal Bureau of Investigation (FBI) James Comey, echoing other citizens and public officials, expressed his dismay at the rudimentary state of statistics on police-initiated civilian deaths.⁴

The FBI’s Supplementary Homicide Reports (SHR) and the National Vital Statistics System (NVSS), the administrative data sources potentially best suited to addressing the topic, are incomplete in recording police violence. The SHR is a voluntary program, and many law enforcement agencies do not participate in it or submit complete reports. Although the NVSS has essentially full coverage, it is vulnerable to measurement errors. In particular, if the death certificates that underlie the NVSS fail to mention police involvement—as frequently occurs—the system will misclassify police-involved deaths as civilian homicides. These problems are pervasive, and they currently make the two systems unsuitable for studying police shootings.¹

A third official data source, the Arrest-Related Deaths program administered by the Bureau of Justice Statistics, collects information on deaths that occur during interactions between police officers and citizens. The Arrest-Related Deaths program historically produced low counts like those from the SHR and NVSS, and its collection efforts are currently in suspension as it undergoes a redesign to improve accuracy.⁵ The specific sources of error in the Arrest-Related Deaths program are not entirely clear, but as with the SHR, they generally stem from incomplete reporting by police agencies.¹

Alternatives to the official systems come from open-source data collections, which are maintained by news media and other private organizations. The open-source collections find many more legal intervention homicides than do the administrative systems.¹ These collections nevertheless require ongoing diligence and uniformity in gathering data if they are to be consistently useful, and their sponsors could discontinue them at any time. By their nature, the open-source collections also lack the authority and credibility that accrue to the products of government statistical agencies. Although they are helpful, open-source data offer only a partial solution to the problems of the administrative systems.

Conner et al. advance the case that NVDRS data can fill the need for accurate counts of fatal police shootings. NVDRS abstractors pool information on violent deaths (including deaths from legal interventions) from death certificates, law enforcement records, and coroner and medical examiner reports. Before 2018, NVDRS had limited national coverage, and Connor et al. analyze data from the 27 states that reported violent death data during 2015. In late 2018, the system expanded to the entire United States, and in the future, it should allow for national estimates.⁶

In a previous evaluation of data quality, Barber et al. compared counts of officer-involved killings from NVDRS, the SHR, and the NVSS in 16 states.⁷ They found that NVDRS counts were roughly 92% higher than those from the SHR and 57% higher than those from the NVSS. The lack of an external standard for accuracy hangs over these comparisons, but as with open-source data, the axiom “Larger is better” seems justified in this case.

In their study, Conner et al. assessed the degree to which NVDRS captures the firearm-related police shootings that appear in several major open-source collections. With a complex labor-intensive procedure, the authors found that NVDRS data can identify almost all (97%) of the incidents that appeared in any of the sources. Perhaps more realistically, the NVDRS abstractor-coded police shootings that researchers might routinely analyze also captured most (93%) of the incidents. Especially considered against the poor performance of the SHR and NVSS, this outcome suggests that NVDRS could serve as the data source on police killings that the nation currently lacks.

Despite the evidence in its favor, NVDRS also has limitations in understanding the nature and scope of fatal police violence. First, the NVDRS counts amount to “good enough” estimates of the number of officer-involved killings. The closest agreement between NVDRS and the open-source data involved an amount of effort that would be unrealistic in practice. The more realistic abstractor-coded data captured fewer of the total incidents than did any of the open-source data sets themselves. Using the NVDRS counts therefore requires a willingness to tolerate some level of bias. Conner et al. note that modifications to NVDRS coding procedures could reduce or eliminate these errors. Yet one could make similar arguments about most errors in the NVSS and, to a lesser degree, the SHR.

The open-source data also contain their own errors, and none of the five collections in the Conner et al. analysis encompass the universe of reported incidents. The handful of cases that Conner and associates found only in NVDRS is a demonstration of at least a small amount of error in the open sources. More problematic, the origin and possible extent of any errors in the open-source data are unknown, and questions about hidden biases are no more than speculations.

Beyond issues of accuracy, the NVDRS counts will not themselves allow the study of many central questions about police violence. By its nature, the system does not include nonfatal injuries sustained in police encounters, and some contextual details that would be useful in understanding police killings are outside its purview. As Conner et al. note, the delays involved in generating NVDRS data also reduce the system’s ability to provide timely information to policymakers.

Overall, NVDRS offers an improvement over the existing administrative data systems in its estimates of fatal police shootings. Errors are common in measuring many criminal justice phenomena, and the accuracy and detail of NVDRS is enough for most practical purposes. NVDRS is not an ideal data source on fatal police shootings, but with its expansion to the entire United States, it should easily be the best source available.

Beyond its relevance to police violence, the Conner et al. study tests the ability of NVDRS to capture an outcome that has proved difficult to quantify. The results of the study support the use of NVDRS for measuring police-involved fatal shootings, but they also offer indirect evidence of NVDRS data quality more generally.

An advantage of NVDRS is its ability to combine multiple independent streams of information, allowing a more complete picture than any single data source can provide. The resulting detail about cases is higher than in the constituent data collections, creating new opportunities for study. The system also provides information on multiple causes of violent deaths, including homicides, suicides, and fatal gun accidents.

Additional tests of NVDRS data quality like the one that Conner et al. undertake would be helpful in further validating uses of the system. With its nationwide expansion, NVDRS should become an increasingly popular source of information, not only about police killings but also about other major categories of violent deaths.