Assessing Data Completeness, Quality, and Representativeness of Justifiable Homicides in the FBI’s Supplementary Homicide Reports: A Research Note

Abstract

Introduction

The most widely used data set for studying police homicides—the Supplementary Homicide Reports (SHR) kept by the Federal Bureau of Investigation—is collected from a voluntary sample.

Materials and Methods

Using a journalist-curated database of police-related deaths, we find the SHR police homicide data to be substantially incomplete. This is due to both non-reporting and substantial under-reporting by agencies. Further, our inquiry discloses a pattern of error in identifying “victims” and “offenders” in the data, and finds that investigating agencies are often incorrectly listed as the responsible agency, which seriously jeopardizes police department-level analyses. Finally, there is evidence of sample bias such that the SHR data system is not representative of all police departments, nor is it representative of large police departments.

Conclusions

We conclude that the SHR data is of dubious value for assessing correlates of police homicides in the United States, as all analyses using it will reflect these widespread biases and significant undercounts. Analysis of SHR data for these purposes should cease.

Introduction

The years since August 9, 2014—the date of the fatal shooting of Michael Brown by Ferguson, Missouri, police officer Darren Wilson—have seen a remarkable increase in scholarly interest in citizen deaths during police activity (see e.g., Barber et al. 2016; Feldman et al. 2017a, b; Kivosto et al. 2017; Shane et al. 2017; Holmes et al. 2019; Conner et al. 2019; Feldman et al. 2019; Hemenway et al. 2019). As scholars have sought to develop a better understanding of the nature, correlates, and possible causes of such deaths, they have run headlong into a major problem: Official, government-collected counts of citizen deaths during police activity (hereafter police-related deaths, or PRDs) are severely inadequate (e.g. Shane and Swenson 2018).

There exist four “official” data sources that include information about PRDs. One is the National Vital Statistics System (NVSS), which is part of the National Center for Health Statistics (NCHS) and includes in its mortality records a category called “deaths by legal intervention” that is intended to capture deaths caused by police action. Two, the Federal Bureau of Investigation (FBI) tracks a specific sort of PRD within its Uniform Crime Reporting (UCR) program via the Supplementary Homicide Report (SHR) program. The UCR has a category called “the killing of a felon by a law enforcement officer in the line of duty” (UCR 2014) that is intended to capture those PRDs caused by intentional police action that result in “justifiable homicide” (see below for more on the FBI’s SHR program and its “justifiable homicide” count methodology). A third data repository that contains information about some aspects of PRDs is the Centers for Disease Control and Prevention’s (CDC) National Violent Death Reporting System (NVDRS). Established in 2002, the NVDRS collects information about various categories of violent deaths in the United States, including those attributed to “legal intervention,” which the CDC defines as deaths caused by law enforcement officers acting in the line of duty. The Bureau of Justice Statistics (BJS) provides a fourth source of PRDs, through its Arrest-Related Deaths (ARD) program. Launched in 2003, but currently suspended due to data quality concerns, the ARD program sought to count citizen deaths that occurred during “all circumstances associated with the actions or events that occurred during an attempt by law enforcement to detain an individual” (Planty et al. 2015).

That the BJS has suspended the ARD program due to data quality concerns speaks to its deficiencies in accurately counting PRDs (Banks and Planty 2015), although recent redesigns integrating several methods have proved promising (Banks et al. 2016). In addition, ARD data collected between 2003 and 2009 are only available for analysis from a restricted data research center in Ann Arbor, Michigan, further hampering its usefulness to researchers.

The other three “official” sources that provide data on some aspect(s) of PRDs have their own limitations. For example, only a small number of states participated in the NVDRS at its 2002 inception point. Consequently, while its scope of coverage has grown over the years to now include all 50 states and the District of Columbia, the fact that many states have only recently begun to participate means the NVDRS provides nationwide counts of death due to legal intervention that are incomplete in even the recent past. For example, Conner et al. (2019) NVDRS-based study of citizens fatally shot by police officers in 2015 contained data from just over half (27) of the states in the Union.

Numerous scholars have documented the liabilities of both the NVSS and SHR with regard to counting police-related deaths. Four decades ago, for example, Sherman and Langworthy (1979) reported that the NVSS undercounted the number of PRDs that it sought to track by as much as 51% nationwide and noted that the FBI didn’t even publish SHR data on felons killed by police due to in-house concerns about data quality. More recently, Loftin et al. (2017, pp. 159) documented the scope and nature of undercounts in both the NVSS and SHR between 1976 and 2013, reporting that “[m]easurement errors were the primary source of underreporting in the NVSS,” while “[c]overage and nonresponse errors were the primary reasons for underreporting in the SHR.”

Despite the documentation in the academic literature of myriad problems with the two longest-running existing official data sources on PRDs, some scholars have pressed forward in recent years with the use of these data—especially the FBI’s SHR counts of “felons killed by the police”—in studies that seek to explain variation in some aspect(s) of PRDs (see, e.g., MacDonald and Parker 2001; Smith 2003, 2004; Holmes et al. 2019). This practice strikes us as potentially problematic, for it risks producing misleading results. While the scholarly warnings about the liabilities of the FBI’s SHR counts of “felons killed by the police” argue that this risk would appear to be non-trivial, we presently have little information about the nature and structure of the errors in official PRD counts, such as those found in the FBI’s SHR.

The purpose of this research note is to shed empirical light on this important debate by taking a deep dive into any potential liabilities of the FBI’s SHR counts of “felons killed by the police.” One aspect of this effort is to examine how and where the SHR undercounts the specific sort of PRD it is designed to count: suspected criminals who are killed by police officers in the line of duty. A second aspect is to examine the extent to which the SHR undercounts PRDs that fall outside of the bounds of the FBI’s definition of “felon killed by the police” (see below). To set the stage for this work, we first provide a brief review of research on deadly police violence that focuses on the data sources used and includes both a sketch of the SHR program and some highlights of critical research. We next talk about an alternative, unofficial database of PRDs that has emerged in recent years that we will use to examine where and how the SHR counts are problematic. This is followed by two sections that report on what our comparative analyses disclosed about the liabilities of SHR counts of what it purports to measure (i.e., “felons killed by the police”) and what the SHR misses with respect to PRDs, respectively. The paper concludes with a discussion of the implications of the findings reported for evaluating extant research and for moving forward with future analyses of the critical matter of citizens dying during police activity.

Research on Police Use of Deadly Force and the Liabilities of the FBI’s SHR Program

Research on the use of deadly force by police officers in the United States has used a variety of data sources and methods to examine numerous questions about the nature and determinants of this ultimate aspect of police violence.¹ One of the earliest empirical studies of deadly force (Robin 1963) focused on fatal shootings by officers in a single police department (Philadelphia) during the 11-year period of 1950–1960, reporting on the frequency of such shootings each year and on various other matters, such as the race, age, and criminal records of the people fatally shot by Philadelphia police officers. Robin also included some data from a group of 9 other agencies during the 1950–1960 time-frame and provided some comparisons across them (and Philadelphia) on matters such as how often officers in the 10 cities killed citizens and the race of the citizens fatally shot. Finally, Robin also provided national counts of persons killed by the police officers from 1950 to 1959 (as reported by the National Office of Vital Statistics; a precursor of the NVSS), and commented on the remarkable stability of the counts from year to year.

Other early works adopted the “small-N” approach pioneered by Robin (1963) in their efforts to examine the nature and determinants of deadly force. Milton et al. (1977), for example, looked at police shootings that resulted in both fatal and non-fatal wounds in seven large U.S. cities during the early 1970s and reported on a variety of issues such as the racial characteristics of the citizens shot and the role that police department policy and shooting review procedures might play in influencing officers’ firearms usage. The tradition of looking at more than just fatal shootings when examining the use of deadly force by police officers was expanded by Fyfe (1978), whose doctoral dissertation looked at various patterns and correlates of all firearms discharges (including those that didn’t strike anyone) by New York City Police Department (NYPD) officers during the years 1971–1975. Included among the main foci of Fyfe’s work (which extended into various post-dissertation publications; e.g., Fyfe (1979, 1980, 1981) were the racial characteristics of both NYPD shooters and citizens taken under fire, the geographic dispersion and concentration of police firearms discharges across the city, and the role that changes in NYPD policies and review procedures played in the temporal patterns of firearms discharges across the five years studied.

As the tradition of examining police use of deadly force using data from single cities and small groups of cities grew in the 1970s, another research tradition also emerged: the examination of patterns and correlates of deadly police violence across the United States. And it was in this tradition that scholars turned their attention to federal-level “official” data on the topic at hand. Kania and Mackey (1977) and Jacobs and Britt (1979), for example, used state-level counts of deaths by legal intervention from the NVSS for the years 1961–1970 to examine a variety of issues, such as the roles that community violence and class conflict might play in determining levels of fatal police violence. Over time, other researchers turned to the SHR data as a measure of fatal police violence. In 1993, for example, Sorensen et al. used city-level SHR counts of justifiable homicide by police officers for the years 1980–1984 to assess the effect that poverty, violent crime, racial composition, and select other factors might have on levels of fatal police violence in U.S. cities with populations of at least 100,000 residents. Since that time, numerous other studies have been published that used SHR data to examine various questions about the determinants of variation in fatal police violence across geographic units, primarily larger cities (e.g., Jacobs and O’Brien 1998; MacDonald and Parker 2001; Smith 2004; Holmes et al. 2019).

The Supplementary Homicide Reports (SHR) is a part of the FBI’s Uniform Crime Reporting (UCR) program that seeks to collect specific information about every homicide investigated by each law enforcement agency that participates in the UCR. But participation in the UCR program is voluntary (e.g., Barnett-Ryan 2006; Loftin et al. 2017). Because of this, and for a variety of other reasons, SHR coverage of homicides in the United States is by no means complete (e.g., Maxfield 1989). Our estimates suggest that in its best years, the SHR represents fatal shootings from law enforcement agencies representing just under 80% of sworn officers and can exclude large state and county non-reporting agencies (e.g., highway patrols and county sheriffs) who are responsible for a substantial number of deaths.

When a law enforcement agency reports having investigated a homicide to the UCR system, the FBI seeks additional data—the age, sex, race, and ethnicity of the decedent and the assailant, the type of weapon used, the relationship between the decedent to the assailant, and the circumstance surrounding the incident—through the SHR program (UCR 2014). One of the circumstance options listed on the Supplementary Homicide Report form is “The killing of a felon by a law enforcement/peace officer in the line of duty” (UCR 2014). Thus, the homicide data collected through the FBI’s SHR should include a count of cases in which American law enforcement officers applied force against a member of the public within the course of their duties and the involved member of the public died as a result of the application of force. But even before scholars turned to the SHR to measure fatal police violence, concerns had been raised about a lack of fit between what the SHR should be measuring, where citizens were killed by police action, and what it actually measures.

Perhaps the first warning about data quality issues with SHR counts of felons killed by police came in 1979 when Sherman and Langworthy reported that the FBI itself possessed “reservations about the quality of those data” because “[m]any police agencies fail to provide some or all of the descriptive information … necessary to discriminate justifiable homicides by police from other forms of homicide” (pp. 547). While Sherman and Langworthy don’t explicitly note this, implicit in their brief discussion of the trouble with the SHR program is the fact that participation in the UCR program in which it resides is voluntary. Quite simply, only agencies that opt to participate in the UCR have the ability to report information about felons killed by the police via the SHR system. And thus, the fact that law enforcement agency participation is non-compulsory, is a major reason for gaps in SHR justifiable homicide counts.

Once scholars began to use the SHR data in studies of fatal police violence, other critiques of the SHR data emerged, critiques that included comparisons of SHR counts with other data sources on police-caused homicides. Fyfe (2002, pp.99), for example, stated that, “when I have compared Justifiable Homicide Report numbers from the FBI with those obtained directly from police departments, I have found virtually no correspondence.” Loftin et al. (2003) assessment of NVSS and SHR data across the U.S. for the years 1976–1998 disclosed that the SHR program grossly undercounted the number of felons killed by the police during those years because some agencies didn’t participate in the UCR program and because some that did participate did not always report to the FBI when their officers killed citizens. Additional evidence of problems with SHR counts of felons killed by the police came from a pair of publications that Klinger (2008, 2012) penned in the wake of Loftin et al. 2003 work. His 2008 piece noted that a comparison of state-by-state SHR police-caused death counts with those found in the aforementioned Deaths in Custody Reporting Program (DCRP) for the years 2003–2005 showed that both programs had serious deficiencies. Klinger’s 2008 piece also started an analysis of problems with SHR counts in the nation’s most populous state that he continued in his 2012 publication. This analysis demonstrated that California SHR data during the years 1996–2008 included notable undercounts of citizens killed by police gunfire both 1) at the state level and 2) for the largest jurisdictions in the state (i.e. the Los Angeles Police Department and the Los Angeles County Sheriff’s Department). More recently, Loftin et al. (2017) extended Loftin et al. (2003) work on identifying problems with SHR counts of felons killed by the police during the years 1976–1998 by adding the years 1999–2013. This work disclosed that the previously identified problems of non- and under- reporting in the SHR continued into the initial years of the twenty-first century. In sum, a host of studies conducted since the late 1970s have established that SHR counts of “felons” killed by the police do not provide a sound representation of the number of citizens who die at the hands of the police in the United States (also see Decker and Pyrooz 2010; Loftin et al. 2015; Renner 2019).

In spite of this decades-long litany of lamentations about liabilities in SHR counts of felons killed by the police, many scholars have insisted on maintaining that the SHR is a sound source of data on citizen deaths at the hands of the police. In hopes of shedding fresh empirical light on the vitally important matter of the adequacy of SHR justifiable homicide counts, we sought to develop additional specific information about the completeness, quality, and representativeness of justifiable homicide counts in the FBI’s Supplementary Homicide Reports program. More specifically, in order to conduct this assessment, our research focuses on three specific research questions:

1. How comprehensive/complete are the SHR counts of justifiable homicides?

2. Are there identifiable, systematic errors in the SHR database?

3. Given that submission to the UCR system is voluntary, how representative are the law enforcement agencies (LEAs) who submit to the SHR of LEAs in the United States?

Data

In order to address these research questions, we employ what is arguably the most comprehensive data collection of police-related deaths in the United States—Fatal Encounters (FE) (Fatal Encounters 2018). FE collected data on n = 23,578 PRDs from 2000 to 2017. Fatal Encounters began with FOIA requests of law-enforcement agencies, combined with news clipping service searches of police homicides in the US. This led to a crowd-sourced effort to identify all incidents of police homicides that simultaneously purged all duplicates from the data set, and now includes regular updates on new incidents and ongoing variable creation for researcher use. A pilot study and ongoing data integration suggest greater coverage than all extant data sets (Finch et al. 2019). In an additional validation study, Conner et al. (2019) finds that Fatal Encounters captured 98.5 percent of a pre-selected number of cases, proportionally more than other open source data sets and the CDC’s National Violent Death Reporting System. Advantages of the FE data include variables indicating specificity in circumstance of death (See Table 1), incident geo-locations, identification of involved police agencies, and near immediate availability of data. Disadvantages include a high rate of missingness for decedent race/ethnicity, and potentially higher rates of missing incidents for earlier years. Ultimately, FE is the largest collection of PRDs in the United States and remains as the most likely data source for historical trend comparisons and police-department-level analyses of the causes of PRDs (Finch et al. 2019).

Table 1.

Circumstance of death counts by year in Supplemental homicide reports and fatal encounters, 2016 Circumstances are noted with the accompanying source category in parentheses; similar circumstances which align in wording or with slight variation in wording by source are listed on the same line. When circumstances are collapsed in one sources but detailed in another, the collapsed category is also listed

Circumstance of death category (Source)	SHR count	SHR%	FE count	FE%
Firearm Circumstances
Firearm, type not stated (SHR)	59	13.44	–
Handgun-pistol, revolver, etc.(SHR)	312	71.07	–
Rifle (SHR)	51	11.62	–
Shotgun (SHR)	5	1.14	–
Other gun (SHR)	4	0.91	–
Gunshot, Intentional Use of Force (FE)	–		1015	63.56
Gunshot, Off-Duty (FE)	–		29	1.82
Gunshot, Suicide (FE)	–		233	14.59
Other Circumstances
Tasered (FE)	–		41	2.57
Knife or cutting instrument (SHR, FE)	2a	0.46	0	0.00
Personal Weapons, including beating (SHR)/ Beaten (FE)	1	0.23	8	0.50
Explosives (SHR)/ Bomb (FE)	1	0.23	1	0.06
Other/type unknown (SHR)/Undetermined, Other (FE)	4	0.91	6	0.38
Blunt Object (SHR)/Bludgeoned w instrument (FE)	0	0.00	1	0.06
Poison-Does not include gas (SHR)	0	0.00	0	0.00
Pushed or thrown out window (SHR)/ Fell from a height (FE)	0	0.00	5	0.31
Fire (SHR, FE)	0	0.00	0	0.00
Narcotics or drugs, sleeping pills (SHR)/ Drug overdose (FE)	0	0.00	5	0.31
Drowning (SHR, FE)	0	0.00	7	0.44
Strangulation-hanging (SHR)	0	0.00	–
Asphyxiation-includes death by gas (SHR)	0	0.00	–
Asphyxiated/Restrained (FE)	–		18	1.13
Chemical agent/Pepper spray (FE)	–		0	0.00
Medical Emergency (FE)	–		17	1.06
Vehicle Circumstances
Vehicle, Accidental (FE)	–		45	2.82
Vehicle, IUF (e.g.PIT maneuvers, spike strips) (FE)	–		6	0.38
Vehicle, Pursuit (FE)	–		155	9.71
Vehicle, Off-Duty (FE)	–		5	0.31
Total	439	100	1597	100

^aTwo incidents coded as “Knife or cutting instrument” in SHR both have Victim 1 and Offender 1 swapped. The second incident incorrectly documents a third party as the victim and does not incorporate officer data at all. The first incident, WI03703-4-1-2016, was the killing of Oswald Mattner, 42, by Officer James Martin of the Wasau Police Department. In this incident, Mattner allegedly pulled a knife on Martin, who shot and killed him on April 30, 2016. The second incident, WI06707-7-1-2016, was the killing of Helmut Wihowski, 58, by Officer Kyle Henning of the Jackson Police Department on July 1, 2016. Wihowski had a knife to his 57-year-old girlfriend’s chest when Henning shot and killed him. The girlfriend did not die. Both incidents are correctly recorded in the Fatal Encounters dataset as Gunshot, Intentional Use of Deadly Force

Comparatively, as previously noted, the Supplementary Homicide Reports (SHR) (United States 2016a) is a database of homicides in the United States maintained by the Federal Bureau of Investigation (FBI) as part of its Uniform Crime Reports program. We specifically focus on Variable 29, which indicates a “felon killed” by police (coded as “81”), or more broadly defined, a justifiable homicide by a law enforcement agent.

To address research question 1, we created an agency-year data file with the 50 largest law enforcement agencies in the United States, based on the number of full-time sworn personnel in the 2008 Census of State and Local Law Enforcement Agencies (CSLLEA) (United States Department of Justice 2008). The data file includes aggregate counts of the number of justifiable homicides reported by each agency to the SHR (NR = not reported to SHR) in a given year. It also included different aggregate counts of the number of police-related-deaths documented in Fatal Encounters (FE) for each agency in a given year. The aggregate FE counts differed by cause of death.

To address research question 2, we relied upon identification of cases within the SHR, limited to the subset of “felons killed by police” as indicated by code 81, of variable 29. We used years 2000–2014 and left the file intact as an incident-level file. We relied upon basic variables such as county and state of incident, as well as month/year, age, and race/ethnicity of “victims.”

To address research question 3, we created an agency-year data file which included FE-based counts of intentional use of force deaths aggregated by agency and year as well as SHR-based counts of justifiable homicides aggregated by agency and year. For the purposes of this study, we restricted our FE-based counts of PRD’s to counts of only non-suicide, intentional use of force PRDs. The dataset included the unique Department Originating Agency Identifier Number (ORI) code for each law enforcement agency as well as Census FIPS county and/or place codes indicating the geographic location of each law enforcement agency. We used these ORI and FIPS codes to merge in additional agency-level and geography-based data sources, depending on the research question (RQ).

We merged in agency-level data on agency size from the 2008 Census of State and Local Law Enforcement Agencies (CSLLEA) (United States Department of Justice 2008), to determine the 50 largest law enforcement agencies in the United States, based on the number of full-time sworn personnel (RQ 1, specifically). Unfortunately, we could not otherwise utilize the CSLLEA or the Law Enforcement Management and Administrative Statistics (LEMAS) as they were sporadically collected and not published after 2013.²

For RQ 3, using a crosswalk file created by the BJS (ICPSR 35,158), we merged in county- and place-level Census data on the socio-demographic characteristics of the lowest level of geography available for each law enforcement agency. We use the decennial census (2000, 2010) and 5-year estimates from the American Community Survey (ACS) (2010–2017) to create basic demographic measures for the geographic locations of law enforcement agencies, including the following measures of interest to our study: proportion of population living below the poverty line, proportion non-Hispanic black, and proportion Hispanic. Data are linearly interpolated between intercensal years.

Using ORI codes and years, we merged in agency- and year-level data from the Law Enforcement Officers Killed and Assaulted (LEOKA) data set (United States 2016b). We used this data to generate agency- and year-level measures of the rate of officers assaulted and the number of officers killed. We discovered substantial missingness in the LEOKA data – 26% of LEA-years in our master dataset were missing LEOKA data for number of officers killed and/or rate of officers assaulted. To prevent 26% of our observations from being dropped from our analysis, we imputed (overall sample) modal values where these indicators were missing. We included a control for these imputed values in our regression models and find that LEOKA missingness is significantly associated with being an incomplete or non-reporter, versus a full SHR reporter. This suggests that dropping observations with missing LEOKA data from our analysis would have generated a sample of LEAs that underrepresented the number and type of LEAs missing from the SHR.

Methods

RQ 1

To examine the comprehensiveness and completeness of the SHR (RQ1), we first describe the ability to identify cause of death in both the SHR and FE. To assess the comprehensiveness of SHR, we selected the 50 largest law enforcement agencies in the United States and compared the total number of justifiable homicides reported to SHR by each agency (NR = not reported to SHR) in a given year to various aggregate counts (by cause of death) of PRD’s reported in the Fatal Encounters data set for each agency in a given year. In some cases, officers from more than one agency are involved, and in these circumstances, we sought to develop information about which agency’s officer(s) were responsible for the death (e.g., whose bullets caused the fatality) by directly contacting each agency. When we were unable to determine that a single agency was responsible, we attributed a partial fraction to each LEA in order not to inflate the total counts in FE. We compare counts overall for the 50 largest LEAs for all causes of death and compare specifically known non-suicide gunshot deaths within those fifty agencies. We also graphically present the entire universe of incidents within SHR and FE to demonstrate representativeness, by cause of death. Finally, as an existence proof, we explored SHR comprehensiveness by looking at what CBC News has identified as “14 High-profile police-related-deaths of U.S. Blacks” (2017) and attempted to ascertain their inclusion/exclusion in the SHR database. Parsing out all “felons killed by police” (item #81 in V29 “OFFENDER 1: CIRCUMSTANCE” of the UCR SHR data), we used the following data points to determine if cases were included: offender age (V23), offender sex (V24), offender race (V25), state (V12), county (V8), year (indicated by annual data set year), month (V13), and agency responsible (V11). Our purpose here, is not to exhaustively document all police-related deaths omitted/included in the SHR, but rather to document among high profile incidents, arguably more apt to be documented, the existence of omitted cases.

RQ 2

To address whether there are identifiable, systematic errors in the SHR database (RQ 2), we examine the extent of missingness in what SHR terms “offender” and “victim” demographic characteristics that may prevent validation and matching with other sources. Given the ambiguity of the terms offender and victim, we examine cases with multiple reported “victims” to determine whether there is evidence of information switching or misclassification of decedent(s) or police officer(s). For two states that have state level independent investigative agencies, we examine whether there is evidence of misclassifying investigating agency versus responsible agency by looking at all state police incidents (either as investigating or responsible agency).

RQ 3

Submission to SHR is voluntary, and many LEAs (including those in the state of Florida) choose not to submit to this system. As such, the submitting agencies are a convenience sample at worst and a non-probabilistic sample at best. At this time, researchers are unable to distinguish whether LEAs that do not submit to the SHR have zero civilian homicides and/or zero JHs, or whether they have non-zero counts of either but choose not to submit to the system. Those who do submit to SHR do so either monthly or annually. This is tracked through the UCR system by noting LEAs that submitted data in any given month. In order to determine reporter status, we link SHR data with UCR data using the unique ORI code and the appropriate year identifier.

We consider those who submitted a report for every month, with a “last reported month” of December, to be UCR “full reporters.” We define those who submitted a report at least one month prior to December to be “incomplete reporters” and those who did not submit a report in any month to be “non-reporters.” Note that a non-reporter status could mean an LEA either had zero homicides to report or chose not to submit to SHR despite having homicides to report. The SHR cannot distinguish between these two cases of non-response. There are no instances in which LEAs participated in SHR but not in UCR.

To assess the degree to which LEAs that report to SHR are representative of the total population of LEAs in the United States, we used a multinomial logistic regression approach to model the predictors of any given LEA’s likelihood of being a full reporter, an incomplete reporter, or a non-reporter in any given year. We selected a multinomial regression model because our outcome of interest is a categorical variable. In the model, full SHR reporters are the omitted category, and we compare full reporters to incomplete reporters and non-reporters. Standard errors are clustered at the level of LEA (based on ORI code) to account for non-independence between agency-years.

At the core of this analysis is an assessment of representativeness that occurs for all national samples that are collected. For example, when a new data set is collected, basic demographics (age, race/ethnicity, gender, educational attainment, e.g.) are compared from the sampled data to population parameters to ensure that the sample is in fact representative. This approach drives our analysis and variable selection.

The following agency-year-level variables are included in our models: logged population size (at lowest available geographic level—place or county), proportion below poverty line (at lowest available geographic level—place or county), proportion non-Hispanic black (lowest geographic unit—place or county), proportion Hispanic (at lowest available geographic level—place or county), greater than zero police-related deaths (based on number of intentional use of force deaths reported in Fatal Encounters), greater than zero officers killed (from LEOKA) greater than zero rate of officers assaulted (from LEOKA: number of officers assaulted divided by total number of officers), and greater than zero police-related deaths with non-white decedents (based on number of intentional use of force deaths with non-white decedents reported in Fatal Encounters).

Results

Research Question 1. How Comprehensive/Complete are the SHR Counts of PRDs?

Conventional wisdom tells us that LEAs generally use the SHR to report incidents of “justifiable homicides” of civilians by police, which are largely fatal gunshots. In the SHR, 98.2% of deaths include an “offender weapon” or gun³; SHR details deaths by type of firearm used with the 71.07% of all deaths specifically by a handgun. Fatal Encounters separates gunshot deaths (79.97% of all deaths reported in FE) by actor instead of weapon, with 63.56% as Officer, Intentional Use of Force, 1.82% Officer, Off-Duty, and 14.59%, as Decedent, Suicide. SHR also reports a series of other “offender weapon categories” most of which have few if any incidents in SHR whereas incidents with these classifications were identified in FE. For instance, SHR identified no incidents in which the category was strangulation or asphyxiation while FE identified 18 incidents in 2016 where the decedent was asphyxiated or otherwise succumbed while being restrained. Notably, in 2016, SHR reports two “knife or cutting instrument” deaths in which they misidentify the victim and offender, what we call a “switching error,” detailed further in the next RQ. The circumstances of the deaths are noted in the Table 1 footnotes, but are not incidents in which officers stabbed decedents, but in fact, gunshot deaths (of decedents with knives) as correctly identified in FE.

Additionally, researchers may be interested in a much broader collection of PRDs which could be amenable to policy changes, such as deaths by other weapons (Tasers, e.g.) as well as deaths that result from officer vehicular accidents—a common, but overlooked phenomenon in explorations of police-related deaths (Frank 2016). Table 1 shows 41 deaths from tasers which are not separated out in SHR. Although vehicular death counts are known to be undercounted in FE (Finch et al. 2019), pursuit deaths are 9.71% of incidents in FE, with an additional 0.38% specifically identified as due to PIT maneuvers or spike strips, and all vehicle-related deaths are 13.22% of all incidents reported by FE in 2016. Pursuit-involved vehicular deaths are virtually unreported in the SHR. Specifically, “offender weapon” in the SHR contains 16 response options for “weapon used” in the homicide, none of which would cover a vehicular death.

Table 2 displays aggregated counts from SHR and FE for the 50 largest LEAs, by LEA and cause of death to further assess completeness. The most striking finding is that 11 of the 50 largest LEAs do not report to the SHR (22%); these non-reporters are largely highway patrol and state police. This could be ignorable if the agencies were not responsible for any PRDs, but our FE comparison notes that these 11 agencies accounted for 34.33 gunshot homicides in 2016 alone, and an additional 12.5 vehicular pursuit deaths (see Table 2). Looking only at gunshot deaths, we find that among those reporting to the SHR database (39 LEAs), 17 (43.6%) under-reported the number of gunshot homicides—this means that 28 of 50 total agencies (56%) under-reported their gunshot homicides as a result of either non-reporting or omission of some documented incidents. This table also highlights heterogeneity in completeness. For instance, looking specifically at known non-suicide gunshot incidents in FE relative to SHR, some agencies reported all of their gunshot homicides (e.g. San Diego PD), some the majority (e.g. Los Angeles PD), while others reported less than half (e.g. San Francisco PD), or none of their gunshot homicides (e.g. Miami-Dade DP) (Table 2).

Table 2.

The 50 largest law enforcement agencies and reporting status in 2016: comparison of SHR versus Fatal Encounters

LEA	FT Sworn personnel	ORI7	SHR counts	FE gunshots	FE IUF deaths	FE Pursuit deaths	FE Suicides and other	FE PRDs total
New York City (NY) Police	36,023	NY03030	9	9	2	2	2	15
Chicago (IL) Police	13,354	ILCPD00	0	11	0	6	1	18
Los Angeles (CA) Police	9727	CA01942	14	19	0	1	1	21
Los Angeles County (CA) Sheriff	9461	CA01900	8	16	1	0	3	20
California Highway Patrol	7202	Multiple	NR	6 + 1@.33	1	9	0	17
Philadelphia (PA) Police Department	6624	PAPEP00	6	5	0	1	1	7
Cook County (IL) Sheriff	5655	IL01600	0	0	0	0	0	0
Houston (TX) Police	5053	TXHPD00	8	7 + 1@.5	0	4	3	15
New York State Police	4847	Multiple	NR	1	0	0	0	1
Pennsylvania State Police	4458	Multiple	NR	5	0	0	2	7
Washington (DC) Metropolitan Police	3742	DCMPD00	7	4	0	1	0	5
Texas Department of Public Safety	3529	TX00000	NR	2 + 3@.5	0	0	1	6
Dallas (TX) Police	3389	TXDPD00	4	4 + 1@.5	1	1	0	7
Phoenix (AZ) Police	3388	AZ00723	17	17 + 1@.5	1	1	1 + 1@.5	22
Miami-Dade (FL) Police	3093	FL01300	NR	5	0	0	0	5
New Jersey State Police	3053	Multiple	NR	1	0	1	0	2
Baltimore (MD) Police	2990	MDBPD00	4	4	0	3	1	8
Las Vegas (NV) Metropolitan Police	2942	NV00201	1	4	0	1	3	8
Nassau County (NY) Police	2732	NY02900	0	0	0	0	0	0
Suffolk County (NY) Police	2622	NY05101	0	0	0	0	0	0
Harris County (TX) Sheriff	2558	TX10100	2	4	0	0	5	9
Massachusetts State Police	2310	Multiple	NR	0	0	0	0	0
Detroit (MI) Police	2250	MI82349	3	0	0	3	1	4
Boston (MA) Police	2181	MAO 1301	0	3	0	0	0	3
Riverside County (CA) Sheriff	2147	CA03300	2	3 + 1@.33	0	0	0	4
Illinois State Police	2105	Multiple	2	2 + 1@.5	0	3 + l@.5	1@.25	8
San Antonio (TX) Police	2020	TXSPD00	8	8	1	3	1 + 1@.5	14
Milwaukee (WI) Police	1987	WIMPD00	3	3	0	0	2	5
San Diego (CA) Police	1951	CA03711	3	3	0	0	0	3
San Francisco (CA) Police	1940	CA03801	1	3	0	0	0	3
Honolulu (HI) Police	1934	HI00200	1	1	0	0	0	1
Baltimore County (MD) Police	1910	MD00301	2	2	1	1	0	4
Columbus (OH) Police	1886	OHCOPOO	0	6	0	1	0	7
Virginia State Police	1873	Multiple	NR	1	0	1	1@.5	3
North Carolina State Highway Patrol	1827	Multiple	NR	3 + 2@.5	0	1@.5	0	6
San Bernardino County (CA) Sheriff	1797	CA03600	1	1	1 + 1@.5	2	0	5
Orange County (CA) Sheriff—Coroner	1794	CA03000	NR	1	0	0	0	1
Michigan State Police	1732	Multiple	NR	1 + 1@.5	1@.5	1	1 + 1@.5	6
Atlanta (GA) Police	1719	GAAPD00	0	1	0	0	0	1
Charlotte—Mecklenburg (NC) Police	1672	NC06001	4	5	0	1	1	7
Port Authority of NY & New Jersey Police	1667	NY03070	NR	0	0	0	0	0
Jacksonville (FL) Sheriff	1662	FL01624	NR	3	1	0	0	4
Broward County (FL) Sheriff	1624	FL00600	NR	1	0	0	0	1
Cleveland (OH) Police	1616	OHCLPOO	0	0	0	1	0	1
Florida Highway Patrol	1606	Multiple	NR	1	0	0	0	1
Indianapolis (IN) Metropolitan Police	1582	INIPD00	2	3	0	0	0	3
Prince George’s County (MD) Police	1578	MD01721	0	2	0	1	0	3
Ohio State Highway Patrol	1560	Multiple	0	0	0	0	3 + 1@.33	4
Memphis (TN) Police	1549	TNMPD00	3	3	0	0	2	5
Denver (CO) Police	1525	CODPDOO	7	5 + 1@.5	0	0	0	6

We present an aggregate assessment of the level of missingness in SHR, compared to FE for all LEAs in the U.S. from 2000 to 2016. These results are presented graphically in Figs. 1, 2, 3. In Fig. 1, we graph all SHR counts from all reporting agencies and compare these numbers with the FE count of gunshot homicides only. Counts are reflected on the left y-axis and ratios of the number of FE counts to SHR counts are identified by the yellow dotted line and quantified on the right y-axis. At the lowest undercount, FE contained 1.5 times more incidents than SHR in 2001, but nearly 2.5 times more incidents in 2016. There is a general tendency toward increasing ratios over time, some of which may be attributable to more comprehensive media reporting and internet record-keeping over time, but the generally matching trends of increases/decreases within each year suggest that while incidents are steadily increasing, so is the level of under-reporting in the SHR. This pattern remains roughly the same when adding an additional 430 homicides from intentional use of force to the 14,484 gunshot homicides documented in FE—as reported in Fig. 2. Casting a wide net and including all PRDs from FE (including vehicular deaths—review Table 1), suggests rampant under-reporting of PRDs in SHR—from a low of 2.5 times more incidents in 2001 to a high of 3.75 times more in 2013.

Fig. 1.

Comparison of incident count in supplementary homicide reports versus Fatal Encounters: gunshots only

Fig. 2.

Comparison of incident count in supplementary homicide reports versus Fatal Encounters: all intentional use of force homicides

Fig. 3.

Comparison of incident count in supplementary homicide reports versus Fatal Encounters: all police-related-deaths

For our existence proof analysis, we found that, of the fourteen decedents noted by CBC News, only three (21.4%) of the incidents were found in the SHR justifiable homicide files (V29, code ‘81′) in the SHR data: Clark, McDole, and Brown (see Table 3). Two additional incidents (Crutcher and Scott) that resulted in officer convictions were coded as “other” (V29, code ‘60′). Nine of the fourteen incidents may have been excluded as they were not judged to be (or not yet judged to be) “justifiable” as the incidents resulted in charges being brought against the officers or a grand jury convened, with seven convictions to date: the cases of DuBose, Gray, Castile, Chapman, Scott, Harris, and Gurley. However, that may not be the only rationale for omission as two incidents entirely omitted from the SHR—Crutcher and Sterling—involved no convictions against officers (“14 high-profile police-related deaths” 2017).

Table 3.

SHR Status of CBC News’ “14 high-profile police-related deaths of U.S. blacks”

Name of decedent	SHR status (V29)	Cause of death	Officer legal disposition (SHR record match variables)
Freddie Gray	Missing	Spinal cord injuries	6 officers charged; 1 hung jury/3 not guilty; 2 charges dropped
Samuel DuBose	Missing	Gunshot	Officer indicted; 2 mistrials, no further trials sought
Philando Castile	Missing	Gunshot	Officer found guilty
Terence Crutcher	‘60’ (Other)	Gunshot	A jury found Officer Betty Jo Shelby not guilty of first-degree manslaughter (A 40-year-old black male in September 2016 by Tulsa PD) SHR ID: OK07205-103-9-2016
Alton Sterling	Missing	Gunshot	No charges brought against officers
Jamar Clark	‘81’	Gunshot	No charges filed (A 24-year-old black male in November 2015 by Minneapolis police) SHR ID: MN02711-1-11-2005
Jeremy McDole	‘81’	Gunshot	No charges brought (A 28-year-old black male in September 2015 in a wheelchair by Wilmington police) SHR ID: DE00206-4-9-2015
William Chapman	Missing	Gunshot	Officer found guilty
Walter Lamar Scott	‘60’ (Other)	Gunshot	Officer found guilty (A 50-year-old black male in April 2015 by North Charleston PD) SHR ID: SC01008-1-4-2015
Eric Harris	Missing	Gunshot	Officer found guilty
Tamir Rice	Missing	Gunshot	Grand jury investigation, no criminal trial
Akai Gurley	Missing	Gunshot	Officer found guilty
Michael Brown	‘81’	Gunshot	Grand jury investigation, no charges filed (An 18-year-old black male in August 2014 by Ferguson police) SHR ID: M009528-1-8-2014
Eric Garner	Missing	Strangulation	Grand jury investigation, no criminal trial

Although SHR does not provide unique identifiers, it is possible to uniquely identify incidents in the data with a combination of four variables: responsible agency ORI7 (V3)-Incident # (V17)-Month of Offense (V13)-Year of Offense (V6)

Finally, it is difficult to determine if the PRDs that appear to be missing in the SHR’s “felons killed by police” might appear elsewhere in the data. For example, a failure to put a Code 81 (Felon killed by police) in V29 would have excluded that death from our analysis. In 2013, out of 12,547 total SHR records, 4,204 were Code 99 (Circumstances undetermined) in V29, 33.5% of the total, which may include some “felons killed by police.” In Table 3 for example, we note two officer-involved shootings (Terence Crutcher and Walter Lamar Scott) that are coded as 60 (other) in the SHR. While the officer who killed Walter Scott pled guilty to federal civil rights charges emanating from Mr. Scott’s death, the Tulsa PD officer who killed Terrance Crutcher, was charged with criminal homicide in state court, but acquitted at trial.

Research Question 2. Are there Identifiable Systematic Errors in the SHR Database?

Noting that SHR data are far from comprehensive and tend to omit clear cases of PRDs, we turn to the question of data quality. One of the problems that we encountered in trying to match incidents in Fatal Encounters with incidents in SHR is that there is higher level of missingness when LEAs report officer demographics, relative to decedent demographics (see Table 4). Age, a highly identifiable variable in combination with several other demographics, is missing for 9.26% of officers, compared with only 0.38% of incidents among decedents. This is also true for the news reports that form the basis of FE—LEAs are often reluctant to release much information to the public about the officers involved. In some jurisdictions, it is illegal to release this demographic data.

Table 4.

Percentage missing for potential matching demographics (2005–2015 SHR)

	Age	Sex	Race	Ethnicity
1st Offender (LEA)	9.26	2.78	4.67	8.50
1st Victim (decedent)	0.38	0.02	1.02	6.33

Note that switching of offender/victim is undetectable in these data

As noted above, the UCR Handbook (UCR 2014) mentions that all decedents should be listed as “victims” while individuals responsible for the homicide should be listed as “offenders.” This is true for civilian homicides, and this pattern is supposed to hold for “felons killed by police” in the SHR data. However, we noted several anomalies in the data that did not appear to match this desired pattern as it suggested that offenders and victims may be switched in some instances. We found one document—the New York State instruction guide for reporting felons killed by police to the state system—which obviously creates confusion with respect to referring to police officers as “victims” (see Fig. 4). Further, it is possible that LEAs would be reluctant to refer to officers as the “offenders” and deceased felons as the “victims,” which, in addition to the noted document, could result in switching errors.

Fig. 4.

NY State police instructions for “justifiable homicides” as officer/victim

We attempted to assess whether these switching errors were common, but as noted above, attempting to identify specific incidents in SHR from FE news reports proved very difficult for several reasons. First, missing data in both FE and SHR made it difficult to determine if cases matched exactly in many instances. Second, when all relevant data was available, the ages and genders of “victims” and “offenders” were often very similar, making it even more difficult to assess whether they had been switched or were just common coding errors and/or discrepancies between data sources.

Given this, we did a search for all incidents from 2005 to 2016 that contained multiple victims since switches in these cases would be much more easily identifiable. We found a total of 46 incidents that involved multiple victims and explored whether they were correct with respect to the identification of victim and offender (i.e., citizen and officer), per the UCR Handbook instructions. We found that 6 of the 46 incidents (13%) were switched (see Table 5) and reversed the identification of victim and offender. Given the layout of the SHR file and given the switches, the most common outcome was for multiple officers to be labeled as “victims” and given their own columns of data entry. We then noted that this led to approximately 18 over-counts in SHR that resulted from switching offender/victim. It is doubtful that switching occurred simply because there were multiple victims. Rather, it is far more likely that switching errors endemic to the nomenclature and would also occur for single victims. Extrapolating from our work on multiple victim cases, therefore, it is not unreasonable to assume that roughly 13% of all incidents of justifiable homicide by police officers catalogued in the SHR contain switching errors.

Table 5.

Switching of Offender and Victim in SHR: 2005–2016

Year	# Multiple victim incidents	# Switched	Overcount	Incident 1 SHR ID	Victim 1 FE ID	Incident 2 SHR ID	Victim 2 FE ID
2005	3	1	3	MT05601-1-8-2005	4475
2006	3	0	0
2007	5	1	2	MI50806-3-11-2007	21,172
2008	2	0	0
2009	3	0	0
2010	4	1	2	AR06003-1-5-2010	9141
2011	4	0	0
2012	4	0	4
2013	4	0	2
2014	3	0	0
2015	5	2	4	AZ00189-1-3-2015	15,866	AZ00705-1-11-2015	15,587
2016	6	1	1	SC02102-1-8-2016	18,012
Total	46	6 (13%)	18

In the case of single officer/decedent incidents, this would have no effect on counts of justifiable homicides. However, a non-trivial rate of switching errors could have profound effects on analyses that utilize victim/offender demographics for analysis, particularly for studies of racial disparity, for example. In addition, we noted two identifiable multiple-victim incidents in which the Victim 2 (V33–V36) was not killed, resulting in two more over-counts in the SHR data for 2007 and 2010. We note that we cannot specify the unique case ID of this incident as there are none in SHR. However, we have developed a system by which unique IDs can be created using four criteria from the SHR: ORI Number-Incident Number-Month-Year. For the 2007 non-fatality overcount, the Unique ID is VA12300-1-12-2007 (Tasha Jennings was the surviving victim). For the 2010 non-fatality overcount, the Unique ID is NJ02004-2-2-2010 (Jaquil Spruill was the surviving victim).

Next, we explored the data in detail with respect to investigating versus responsible agencies, given that many LEAs do not conduct investigations of homicides by their own officers, and many agencies do not investigate homicides at all. Two states that have higher-level investigative units are Virginia and New Jersey—both of which often rely on various state police posts to investigate officer-involved shootings. We looked at seven PRD incidents which contained “SP” suffixes in these two states during 2016, indicating state police agencies, but with unique ORIs that related to the local posts. The first incident in NJ (NJ003SP-2-7-2016) was one in which a decedent was killed by a SWAT team in Ocean County. While it is possible that this officer was a member of the state police, this is entirely unclear in the SHR data. The second NJ incident (NJ015SP-1-2-2016) was one in which the decedent was killed by the Ocean County Regional SWAT Team. The officer(s) responsible could have been a member of the NJ State Police or the Ocean County PD, but the responsible agency remains unclear. For the five incidents in Virginia, on the other hand, it is unambiguous that the reporting agency (VA State Police) does not employ the responsible officer, and therefore, the responsible agency is clearly misidentified.

The first VA incident (VA015SP-1-2-2016) was one in which the decedent was killed by Buckingham County deputies, but the Buckingham County post of the VA State Police (SP) is the agency of record. The remaining incidents contain similar mischaracterizations of the responsible agency (VA113SP-1-2-2016; VA053SP-1-8-2016; VA117SP-1-4-2016; VA122SP-2-3-2016). The second through fifth VA incidents were investigated (and reflect the subsequent ORI) by VA State Police posts in Hopewell, Loudoun County, Norfolk, and Richmond. The responsible agencies were actually the Hopewell PD, the Loudoun County Sheriff’s office, the Loudon County Sheriff’s office, and a Richmond-based SWAT team, respectively. Thus, while SP posts are geographically similar, SHR erroneously lists the investigating agencies and not the responsible agencies. This has enormous implications for studies at the agency level.

Research Question 3. Given that Submission to the UCR System is Voluntary, How Representative are the Law Enforcement Agencies (LEAs) Who Submit to the SHR Data of LEAs in the United States?

Of the 233,655 city LEA years in our dataset, 68.68% are full reporters, 3.74% are incomplete reporters, and 27.59% are non-reporters. Of the 5015 large-city LEA years in our dataset, 97.27% are full reporters, 0.64% are incomplete reporters, and 2.09% are non-reporters.

Tables 6 and 7 present results from our multinomial logistic regression comparing full reporter (omitted category), incomplete reporter, and non-reporter LEAs. Significant coefficients indicate SHR data non-representativeness with respect to the variable specified. Statistically significant interaction terms between PRD counts (aggregate counts of PRD’s) and other variables suggest that researchers who engage in associational analysis of each respective variable can expect biased regression coefficients. If all terms are non-significant, then regression results will largely be unbiased and analyses of the SHR could proceed without this concern.

Table 6.

All Local City Police Departments, 2000–2016

	Incomp reporter (vs. full reporter)			Non-reporter (vs. full reporter)
	(1)	(2)	(3)	(4)	(5)	(6)
Population size (logged)	− 0.23*** (0.01)	− 0.23*** (0.01)	− 0.23*** (0.01)	− 0.19*** (0.01)	− 0.19*** (0.01)	− 0.19*** (0.01)
Prop. below poverty line	1.28*** (0.22)	1.28*** (0.22)	1.28*** (0.22)	0.69** (0.23)	0.69** (0.23)	0.69** (0.23)
Prop. non-hispanic black	0.72*** (0.10)	0.72*** (0.10)	0.71*** (0.10)	0.37** (0.12)	0.37** (0.12)	0.36** (0.12)
Prop. Hispanic	− 2.23*** (0.21)	− 2.23*** (0.21)	− 2.23*** (0.21)	− 1.47*** (0.18)	− 1.47*** (0.18)	− 1.47*** (0.18)
> 0 Police-related deaths (FE)	− 0.34** (0.12)	− 0.31 (0.17)	− 0.43** (0.16)	− 0.63*** (0.08)	− 0.78*** (0.11)	− 0.79*** (0.10)
# of Police-related deaths (FE)
> 0 Officers killed	− 0.91 (0.51)	− 0.91 (0.51)	− 0.93 (0.51)	− 3.16*** (0.76)	− 3.16*** (0.76)	− 3.17*** (0.76)
# Officers killed
> 0 Rate of officers assaulted	− 1.36*** (0.04)	0.26*** (0.01)	− 1.36*** (0.04)	− 6.29*** (0.32)	− 6.29*** (0.32)	− 6.29*** (0.32)
Rate of officers assaulted
> 0 PRD w/ non-white decedent		− 0.05 (0.24)			0.27 (0.15)
> 0 PRD X Prop. Non-Hisp Black			0.47 (0.46)			0.83* (0.37)
Imputation flaga	Yes	Yes	Yes	Yes	Yes	Yes
Constant	− 0.70*** (0.11)	− 0.70*** (0.11)	− 0.70*** (0.11)	0.80*** (0.11)	0.80*** (0.11)	0.81*** (0.11)
Observationsb	233,665	233,665	233,665	233,665	233,665	233,665

^***p < 0.001;

^**p < 0.01;

^*p < 0.05

^aImputaton flag identifies observations with missing LEOKA data for officers killed and/or rate of officers assaulted. Modal values of 0 were imputed for these observations

^bEach observation is a department and year

Table 7.

Largest city police departments (25 k population or more), 2000–2016

	Incomp reporter (vs. Full reporter)			Non-reporter (vs. Full reporter)
	(1)	(2)	(3)	(4)	(5)	(6)
Population size (logged)	− 0.83 (0.49)	− 0.87 (0.50)	− 0.81 (0.48)	1.12*** (0.32)	1.16*** (0.31)	1.12*** (0.31)
Prop. below poverty line	− 3.59 (4.23)	− 3.55 (4.21)	− 3.72 (4.20)	− 31.52* (13.79)	− 31.00* (13.38)	− 31.47* (13.46)
Prop. non-hispanic black	2.91* (1.23)	2.69* (1.23)	4.16* (1.62)	8.79** (3.12)	8.78** (3.05)	9.56** (3.01)
Prop. hispanic	− 0.01 (1.13)	− 0.18 (1.12)	− 0.08 (1.13)	− 0.54 (4.04)	− 0.45 (4.06)	− 0.76 (4.09)
> 0 Police-related deaths (FE)	0.38 (0.40)	− 0.27 (0.73)	0.89 (0.56)	− 0.94 (0.69)	0.12 (0.57)	0.29 (0.62)
# of Police-related deaths (FE)
> 0 Officers killed	− 0.19 (1.07)	− 0.21 (1.07)	− 0.16 (1.07)	1.26 (1.89)	1.33 (1.87)	1.35 (1.83)
# Officers killed
> 0 Rate of officers assaulted	− 1.18** (0.37)	− 1.17** (0.37)	− 1.24** (0.39)	− 3.59* (1.58)	− 3.63* (1.58)	− 3.72* (1.61)
Rate of officers assaulted
> 0 PRD w/Non-white decedent		0.85 (0.82)			− 1.36 (0.85)
> 0 PRD X Prop. Non-Hisp Black			− 2.33 (1.83)			− 4.59* (2.13)
Imputation flaga	Yes	Yes	Yes	Yes	Yes	Yes
Constant	5.86 (5.66)	6.41 (5.74)	5.45 (5.62)	− 4.72*** (3.88)	− 15.22*** (3.77)	− 14.82*** (3.78)
Observationsb	5,015	5,015	5,015	5,015	5,015	5,015

^***p < 0.001;

^**p < 0.01;

^*p < 0.05

^aImputaton flag identifies observations with missing LEOKA data for officers killed and/or rate of officers assaulted. Modal values of 0 were imputed for these observations

^bEach observation is a department and year

All City Police Departments

Table 7 reports results for all local city police departments and years 2000–2016. In models not shown, we estimated the independent, unadjusted relationship between the individual department characteristics included in Table 6 and SHR reporting status in a given year.

With respect to incomplete reporting, we find that police departments located in places or counties with higher levels of poverty and a higher proportion of non-Hispanic blacks are statistically significantly more likely than departments located in places with lower levels of poverty and a smaller non-Hispanic black population to be incomplete SHR reporters versus full SHR reporters. On the other hand, we find that police departments located in places or counties with higher population sizes and a larger proportion of Hispanics are statistically significantly less likely to be incomplete SHR reporters than full reporters. We also find that police departments with a nonzero number of PRDs, nonzero number of officers killed, or a nonzero rate of officers assaulted are statistically significantly less likely to be incomplete SHR reporters than full reporters.

With respect to non-reporting, we find that police departments with a higher number of officers killed and departments located in places or counties with higher levels of poverty and a higher proportion of Hispanics are statistically significantly more likely than other departments to be non-reporters versus full SHR reporters. We also find that departments regional population size and proportion of non-Hispanic blacks, as well as department characteristics, including number of PRDs, nonzero number of officers killed, and rate of officers assaulted, to have a statistically significant negative association with the likelihood of non-reporting.

In models 1 and 4, we estimate the adjusted relationship between these police department characteristics and reporting status. With respect to incomplete reporting (model 1), we find that all the associations noted above hold their direction and remain statistically significant except for the association between number of officers killed and relative odds of incomplete reporting, which becomes non-significant. With respect to non-reporting (model 4), we find that all of the associations noted above hold their direction and remain statistically significant except for the association between proportion non-Hispanic black and relative odds of non-reporting, which becomes statistically significantly positive. This indicates that, when other department characteristics are adjusted for, departments located in places or counties with a higher proportion of non-Hispanic blacks are in fact statistically significantly more likely to be non-reporters than departments with a lower proportion of non-Hispanic blacks.

In models 2, 3, 5, and 6, we explore whether the relationship between number of PRDs and reporting status varies according to the number of non-white decedents or the size of the non-Hispanic black population. Our results in models 2 and 5 indicate that adjusting for the number of non-white decedents does not account for the association between number of PRDs and reporting status. Our results in models 3 and 6 indicate that the negative association between number of PRDs and relative odds of non-reporting is reduced in areas with a relatively higher NH Black population.

Altogether, these results indicate that there are systematic biases in the types of city police departments that report justifiable homicides by their officers to SHR. Notably, police departments with at least one PRD are more likely to report to the system than departments with zero PRDs, which biases SHR results toward representing departments with a nonzero number of police homicides. Our findings also suggest that the SHR overrepresents police departments with higher rates of officers assaulted and departments located in places or counties that are more populous or have a high proportion of black residents. The SHR underrepresents police departments with high numbers of officers killed and departments located in areas with high rates of poverty and large Hispanic populations.

“Large” City Police Departments

In Table 7, we run the same set of models but restrict our sample to police departments located in cities with a population of 25,000 or more. In unadjusted models not shown, we find that the rate of officers assaulted in a department and the regional proportion of non-Hispanic blacks are negatively associated with incomplete SHR reporting. With respect to non-reporting, we observe a wider range of associations. We find that population size is positively associated with non-reporting while poverty, proportion Hispanic, a non-zero number of PRDs, and a non-zero rate of officers assaulted are negatively associated with non-reporting.

In models 1 and 4, we estimate the adjusted relationship between these police department characteristics and reporting status. With respect to incomplete reporting, we find that the coefficient for rate of officers assaulted holds its direction and significance, indicating that police departments with a non-zero rate of officers assaulted are statistically significantly more likely than other departments to be incomplete versus full SHR reporters. However, the coefficient for proportion of non-Hispanic blacks changes direction and becomes positive, indicating that when other characteristics are adjusted for, departments located in places or counties with a higher proportion of non-Hispanic blacks are significantly more likely than other departments to be incomplete versus full SHR reporters.

With respect to non-reporting, we find that the coefficients for population size, city poverty, and rate of officers assaulted hold the association and significance noted above. The association between proportion non-Hispanic black and relative odds of non-reporting becomes statistically significant and positive, departments located in places or counties with a higher proportion of non-Hispanic blacks are significantly more likely than other departments to be non-reporters versus full SHR reporters.

In models 2, 3, 5, and 6, we test whether the relationship between PRDs and reporting status varies according to the number of non-white deaths or the size of the non-Hispanic black population as well as. Our results in model 6 indicate that the negative association between number of PRDs and relative odds of non-reporting is reduced in areas with a relatively higher non-Hispanic black population. We do not observe any other statistically significant associations or interactions.

Altogether, these results indicate that there are systematic biases in the types of large city police departments that report to SHR. Our results suggest that the SHR overrepresents large police departments with a nonzero rate of officers assaulted as well as large departments located in places or counties with high rates of poverty. Furthermore, the SHR underrepresents large police departments located in more populous regions and regions with large non-Hispanic black populations. In addition, we find that large police departments with a nonzero number of PRDs are more likely to report to the system and be represented in SHR data if they are located in places or counties with a higher proportion of non-Hispanic black residents.

Discussion and Conclusions

Using a journalist-curated database of police-related deaths (police homicides plus other types of arrest-related-deaths) as a comparison data set, we find the law enforcement Justifiable Homicide data kept by the FBI’s SHR program to be functionally incomplete. By design, it excludes all police-related deaths that are not deemed “justifiable” police homicides, and it further excludes a large proportion of police homicides, including many high-profile incidents (as illustrated by our evidence proof of fourteen high-profile cases). These selective incidents provide clear indication that the SHR is failing to collect even high-profile incidents, possibly because LEAs may not consider these to be “justifiable” due to ongoing legal cases or ultimate judicial disposition, or because decedents may have not been engaged in felonious activity. Regardless of the cause of death or judicial disposition of the homicide, as researchers, a larger set of PRDs than is contained in the SHR is preferred and necessary for thorough analyses. Undercounts and under-coverage will affect studies that aggregate agencies and those that conduct department-level analyses. Based on a small case study, it is clear that incident submission to SHR is fairly haphazard and may not include a significant number of PRDs that appear to be relevant to data collection goals.

Further, there is a non-trivial amount of switching errors in erroneously defining “victims” and “offenders” in the data, leading to marginal overcounts, which further compromises the comprehensiveness of the data. In addition, for those seeking to do department-level analyses, investigating agencies may be determined to be the responsible agency, when in many cases they are not. Misidentification of the investigating agency as the responsible agency may not affect analyses which are aggregated to counties or states, for example, but will absolutely affect agency-level analyses. Switching errors on the other hand, will affect all types of analyses, will possibly inflate incident numbers for some agencies, and will seriously threaten analyses that use victim/offender demographics.

Finally, there is clear evidence of sample bias such that the SHR data system is not representative of all city police departments nor all large city police departments. Our conclusion is that we find the data set to be unusable for assessing correlates of police-related deaths in the United States, as all extant analyses will reflect these widespread biases. It is unknown as to whether non-representativeness will affect aggregate-level analyses, but it is clear that this will be severely detrimental to both generalizability and statistical inference in department-level analyses. More than likely, even aggregate-level analyses such as metropolitan or county-level analyses will be biased in similar directions, although we do not directly examine this matter in our research.

Because the SHR lacks comprehensiveness and accuracy, it is not sufficient for the analysis of PRDs (police homicide or any other type of police-involved death). At the same time, we are not stipulating that the FE dataset is sufficient by itself for the analysis of PRDs, only that it is the most accurate and comprehensive that currently exists. Justifiable homicides are just a small part of the total number of deaths that result from police interaction, and so, by relying on official data sources such as the SHR, the current literature fails to represent and understand the true prevalence, determinants, and true societal impact of police-related deaths in the U.S.

On another note, we argue that some of the critiques levied here may not apply to a newly funded data collection effort by the FBI—the National Use of Force (NUF) Data Collection. However, several of the SHR system critiques will likely still apply. First, the NUF remains a voluntary data program as the FBI has no legal authority to mandate participation in it by any law enforcement agency. Thus, it is highly likely that this data set will remain a convenience sample and biased toward agencies with characteristics similar to what we outline above. This will make statistical inference to a known population virtually impossible, and it will limit generalizability of the results beyond the voluntary sample. Second, missing data among those agencies that do opt to participate will likely remain a problem in the NUF, as virtually every data entry field allows for a “pending” designation, including characteristics of the officers and subjects. Finally, incorrect identification of responsible agencies may well persist in the NUF, and under-reporting of incidents will likely remain rampant. On the up side, as the NUF seeks to collect information on forceful police actions that do not result in death—including non-fatal weapon discharges and non-fatal injuries from other police actions—it could prove to be a major step forward in documenting police use of force.

Finally, whatever the merits of fuller use of force reporting might turn out to be, the current paper has demonstrated that present counts of police-involved deaths kept by the FBI are highly problematic and unsuitable for analysis of prevalence or correlates of police homicides and other police-related deaths. Based on our analyses, the blind use of SHR to provide incidence/prevalence estimates of police homicides should cease, and we further argue that analyses of the correlates/causes of police homicides should not be based on SHR data. Data quality has been proven to be very poor overall, despite the possible accuracy of data for particular law enforcement agencies. Further, unless and until careful assessments demonstrate that the FBI’s NUF program does a sound job of collecting the data on forceful police actions that it seeks to gather, researchers should be wary of using this new data source once it becomes available. We believe that the 80% threshold set by the FBI will be very difficult to meet. Regardless, there is sound reason to suspect that many of the problems that plague the SHR program will also be present in the NUF data base.