SOCIAL SCIENCES Correction for “Procedural justice training reduces police use of force and complaints against officers,” by George Wood, Tom R. Tyler, and Andrew V. Papachristos, which was first published April 20, 2020; 10.1073/pnas.1920671117 (Proc. Natl. Acad. Sci. U.S.A. 117, 9815–9821).
The authors wish to note the following: “The analysis contains a subtle statistical issue that warrants a reanalysis of the data using a different methodology. The outcome variables in the analysis are the frequency of complaints, sustained or settled complaints, and use of force by cluster-month. The analysis used the trend in the outcomes among later-trained clusters to estimate the counterfactual outcomes for earlier-trained clusters. However, later-trained clusters consist of fewer officers than earlier-trained clusters and the outcomes are in secular decline throughout the study period. Consisting of fewer officers, the later-trained clusters undergo a relatively small decrease in the frequency of the outcomes due to the secular decline. The counterfactual outcomes for the early-trained clusters are guided by this relatively small decrease. However, we should expect the counterfactual outcomes to undergo a larger secular decline among early-trained clusters because these clusters contain more officers, on average.
“This issue was documented and brought to our attention by Jonathan Roth and Pedro H.C. Sant’Anna. A reanalysis of the data, produced in collaboration with Roth and Sant’Anna, is available online at: https://osf.io/preprints/socarxiv/xf32m/ (1). The reanalysis uses officer-level outcomes and the estimation procedure in Callaway and Sant’Anna (2). With data made available since publication, we also extended the panel by an additional 9 mo to cover the entire roll-out of training. Data and code for the reanalysis is available at: https://github.com/george-wood/procedural_justice_revisited.
“The reanalysis provides evidence that training reduced use of force, but with some uncertainty around the magnitude of the effect. Officers reported using force 0.047 times per month in the 12 mo before training, an average of 363 uses of force per month across the 7,785 officers. The simple weighted average ATT (SWATT) is -0.011 (95% CI: -0.020, -0.001) uses of force per officer month. At the low end, the reduction of force would result in 8 fewer uses of force per month across all officers, while at the high end the average reduction would result in 156 fewer uses of force per month.
“The reanalysis does not provide statistically significant evidence of an effect on complaints or sustained complaints. Officers received 0.044 complaints per month in the 12 mo before training. The SWATT is −0.005 (95% CI: −0.013, 0.003) complaints per officer-month. Although the magnitude of this effect is meaningful, given the uncertainty in the estimate it would be unwarranted to rule out the possibility that training had no effect on complaints. The SWATT is near zero for sustained and settled complaints (95% CI: −0.002, 0.002).” The corrected Fig. 2 and its legend, based on the reanalysis and updated data, appears below.
Fig. 2.
(Top) Observed and counterfactual estimates of complaints, sustained or settled complaints, and use of force per 100 officers per month. Months are relative to the onset of training. The counterfactual estimate is the dynamic ATT subtracted from the observed outcome. The estimates are based on the reanalysis using the procedure in Callaway and Sant’Anna (2) and the extended panel data. (Bottom) Dynamic ATT per officer by month relative to training. Monthly ATT estimates are colored according to their value relative to zero. 95% pointwise confidence intervals are shown in blue and 95% simultaneous confidence intervals, which account for multiple-hypothesis testing, are shown in gray.
1. G. Wood, T. R. Tyler, A. V. Papachristos, J. Roth, P. H. C. Sant’Anna, Revised findings for “procedural justice training reduces police use of force and complaints against officers.” Available at SocArXiv: https://osf.io/preprints/socarxiv/xf32m/ (2021).
2. B. Callaway, P. H. C. Sant’Anna, Difference-in-differences with multiple time periods. https://ssrn.com/abstract=3148250 Econom. J., in press.