Making inferences about racial disparities in police violence

A recent PNAS study, Johnson et al. (1), investigates the role of race in fatal police shootings. Unlike previous studies which focused on victim race alone, the paper features original data about the race of officers who use deadly force and offers a rare accounting of other shooting attributes that contextualize fatal encounters. Johnson et al. (1) discuss possible “discrimination by White officers” (ref. 1, p. 15877), but conclude racial diversity in police agencies brings limited benefits—a claim cited by major news outlets and in US Congressional testimony, inflaming an already contentious policy debate.

Despite the value of this much-needed research, its approach is mathematically incapable of supporting its central claims. In this letter, we clarify the gap between what Johnson et al.’s study asserts and what it actually estimates, as well as the implications of that difference for policymaking and future scholarship on race and policing.

Johnson et al.'s study asks “the degree to which Black civilians are more likely to be fatally shot than White civilians” (ref. 1, p. 15877) and prominently asserts “White officers are not more likely to shoot minority civilians than non-White officers” (ref. 1, p. 15877). In the language of probability, Johnson et al.’s study (1) concludes

$$\mathrm{Pr}(\mathrm{shot}\mid \text{minority civilian},\text{white officer},X)\hspace{1em}\hspace{1em}-\mathrm{Pr}(\text{shot}\mid \text{minority civilian},\text{minority officer},X)\le 0,$$ [1]

where $X$ are encounter attributes.

Johnson et al.’s (1) analysis cannot recover these shooting rates because all observations in the data involve shootings. Instead, it estimates “whether a person fatally shot was more likely to be Black (or Hispanic) than White” (ref. 1, p. 15880), which does not correspond to the stated assertions. In a preprint response to our concerns, Johnson and Cesario (2) acknowledge the gap between the claim and the quantity estimated. Yet despite this, Johnson et al.’s (1) original paper infers no “evidence of anti-Black or anti-Hispanic disparity…and, if anything, found anti-White disparities” (ref. 1, p. 15880) simply because more fatally shot civilians are White.*

Johnson et al.’s (1) analysis cannot inform the original claims without accounting for Bayes’ rule:

$$\begin{array}{ll}\hfill & \mathrm{Pr}(\text{shot}\mid \text{minority civilian},\text{white officer},X)\hfill \\ \hfill & \hspace{1em}-\mathrm{Pr}(\text{shot}\mid \text{minority civilian},\text{minority officer},X)\hfill \\ \hfill & \hspace{1em}\hspace{1em}=\frac{\mathrm{Pr}(\mathrm{m}\mathrm{i}\mathrm{n}.\mathrm{c}\mathrm{i}\mathrm{v}.\mid \mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t},\text{white off}.,X)\times \mathrm{Pr}(\text{shot}\mid \text{White off}.,X)}{\mathrm{Pr}(\text{minority civilian}\mid \text{White officer},X)}\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{\mathrm{Pr}(\mathrm{m}\mathrm{i}\mathrm{n}.\mathrm{c}\mathrm{i}\mathrm{v}.\mid \mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t},\text{min}.\text{off}.,X)\times \mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \text{min}.\text{off}.,X)}{\mathrm{Pr}(\text{minority civilian}\mid \text{Minority officer},X)}.\hfill \end{array}$$ [2]

Johnson et al.’s (1) study examines only part of the numerators in Eq. 2’s right-hand side, terms dealing with $\mathrm{Pr}(\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n}\mid \mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t},\dots )$. Because it does not consider how many minority or White civilians are encountered, $\mathrm{Pr}(\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n}\mid \dots )$—Eq. 2’s denominators— Johnson et al.’s (1) study does not show whether “Black civilians are more likely to be fatally shot than White civilians” (ref. 1, p. 15877); i.e., $\mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \mathrm{b}\mathrm{l}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n},\dots )>\mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \mathrm{w}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n},\dots )$. Similarly, the claim that “White officers are not more likely to shoot minority civilians than non-White officers” (ref. 1, p. 15877), i.e., $\mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n},\text{White officer},X)\le \mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{c}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{a}\mathrm{n},\mathrm{n}\mathrm{o}\mathrm{n}-\text{White officer},X)$, is unsupported. The omission of $\mathrm{Pr}(\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{t}\mid \text{Officer race},X)$—the second part of Eq. 2’s numerators—further separates the stated claim and the quantity estimated. As Eq. 2 makes clear, the addition of controls, X, such as the number of crimes committed by each racial group, does not solve these conceptual issues.

Johnson et al.’s (1) study describes attributes of fatal police shootings. While a contribution, these facts alone cannot inform the relative likelihood of White and non-White officers shooting racial minorities. Readers and policymakers should keep this important limitation in mind when considering this work.