Abstract
Objectives. To assess the validity of demographic data reported in news media–based data sets for persons killed by police in Massachusetts (2004–2016) and to evaluate misclassification of these deaths in vital statistics mortality data.
Methods. We identified 84 deaths resulting from police intervention in 4 news media–based data sources (WGBH News, Fatal Encounters, The Guardian, and The Washington Post) and, via record linkage, conducted matched-pair analyses with the Massachusetts mortality data.
Results. Compared with death certificates, there was near-perfect correlation for age in all sources (Pearson r > 0.99) and perfect concordance for gender. Agreement for race/ethnicity ranged from perfect (The Counted and The Washington Post) to high (Fatal Encounters Cohen’s κ = 0.92). Among the 78 decedents for whom finalized International Classification of Diseases, 10th Revision (ICD-10), codes were available, 59 (75.6%) were properly classified as “deaths due to legal intervention.”
Conclusions. In Massachusetts, the 4 media-based sources on persons killed by police provide valid demographic data. Misclassification of deaths due to legal intervention in the mortality data does, however, remain a problem. Replication of the study in other states and nationally is warranted.
US government agencies tasked with monitoring killings by police underreport the number of these deaths at high rates.1,2 Evidence indicates that National Vital Statistics System mortality data, maintained by the Centers for Disease Control and Prevention, are no exception.3,4 For decedents killed by law enforcement, mortality records are assigned a diagnostic code, under the International Classification of Diseases, 10th Revision (ICD-10; Geneva, Switzerland: World Health Organization; 1992), indicating “legal intervention,” defined as “injuries inflicted by the police or other law-enforcing agents, including military on duty, in the course of arresting or attempting to arrest lawbreakers, suppressing disturbances, maintaining order, and other legal action.” These deaths are misclassified if assigned a different code.
News media reports provide an alternative source of national data to identify legal intervention deaths, but the validity of their demographic data has not been established. We conducted novel analyses to (1) quantify the proportion of misclassified mortality records among the legal intervention deaths identified through media-based data sources and (2) assess the validity of demographic data in media sources compared with death certificates, treating the latter as a “gold standard” (as per previous death certificate validity studies5,6).
METHODS
We used Massachusetts as a case study and created a list of the state’s legal intervention deaths from January 1, 2004, to April 14, 2016, according to 4 news media–based data sources: (1) WGBH police-involved deaths (January 1, 2004–December 31, 2014)7; (2) Fatal Encounters (January 1, 2004–April 14, 2016)8; (3) The Guardian’s The Counted (January 1, 2015–April 14, 2016)9; and (4) The Washington Post Fatal Force (online police shooting database: firearm deaths only; January 1, 2015–April 14, 2016).10 Each database provides information about decedent’s name, age, gender, race/ethnicity, and circumstances of death.
WGBH News, a Boston, Massachusetts, public media network, identified deaths through a review of media and law enforcement reports.7 Fatal Encounters, a nonprofit organization, had staff confirm details of crowd-sourced reports.8 The Guardian and The Washington Post newspapers identified deaths through various media sources and confirmed details with investigative reporting.9,10
We reviewed deaths reported in these 4 sources and included only those consistent with the definition of legal intervention, which is not contingent on the lawfulness or intent of the killing. For excluded cases, we documented the reason that they did not conform to this definition (Table A; available as a supplement to the online version of this article at http://www.ajph.org). For included decedents, we retrieved death certificates and ICD-10 codes from the Massachusetts Registry of Vital Records and Statistics, which are publicly accessible in Massachusetts.11 If a legal intervention death was not reported by any of the media-based sources, then it was not included in our analyses.
Funeral home directors provide demographic information on death certificates. The National Center for Health Statistics, relying on a computer program and trained nosologists, assigns an underlying cause of death code based on death certificate literal text (e.g., text written in the “describe how the injury occurred” field). To assess ICD-10 misclassification for the deaths identified through news media, we calculated the proportion of media-identified fatalities for which the mortality record was not coded as legal intervention (codes Y35.0–Y35.7 or Y89.0, excluding legal execution: Y35.5) overall, by year, and by death mechanism (firearm vs other). We also conducted matched-pair analyses to quantify discordance between demographic variables reported on death certificates and each media-based data source, calculating Pearson r for age and Cohen’s κ for gender and race/ethnicity. Because 2 data sources, WGBH and Fatal Encounters, had high levels of missingness for race/ethnicity (16.2% and 31.0%, respectively), we used the Fisher exact test for the null hypothesis that these values were missing at random.
RESULTS
After removing 16 cases from the media sources not meeting the criteria for legal intervention (Table A), we identified 84 civilians killed by police in Massachusetts over the study period. We located death certificates for all decedents. The 6 most recent decedents did not have finalized ICD-10 codes, and we excluded them from the analysis. Of the 78 remaining, 59 (75.6%) were properly coded as legal intervention deaths in vital statistics data. The proportion properly classified was 50.0% for 2004 to 2006, 75.0% for both 2007 to 2009 and 2010 to 2012, and 88.5% for 2013 to 2015 (Figure A; available as a supplement to the online version of this article at http://www.ajph.org). Among misclassified deaths, legal intervention was reported as another assault-related diagnosis for 16 of 19 cases. All 4 of the non-firearm-related cases were misclassified (1 Taser-related, 1 vehicle-related, and 2 medical neglect– related incidents). Most cases (68 of 78; 87.2%) had death certificates with reference to police (typically, “shot by police”); of these, 59 of 68 (86.8%) received proper ICD-10 codes. For 10 cases, no mention of police was found on death certificates, and all were misclassified.
Data on gender for all 4 news media–based sources were perfectly concordant with death certificate values (Table 1). There was near-perfect correlation (Pearson r > 0.99) between the ages listed in each journalistic source and those on death certificates. Race/ethnicity values were perfectly concordant relative to death certificates for The Counted and The Washington Post. Validity for race/ethnicity in the other 2 sources was excellent (Fatal Encounters Cohen’s κ = 0.92; 95% confidence interval [CI] = 0.73, 1.00; WGBH Cohen’s κ = 0.94; 95% CI = 0.74, 1.00). The Fisher exact test failed to reject the null hypothesis that missingness was independent of race/ethnicity for WGBH (P = .72) and Fatal Encounters (P = .45).
TABLE 1—
Summary Statistics and Reliability of Sociodemographic Data From 4 News Media–Based Data Sources Compared With Death Certificates for Persons Killed by Police: Massachusetts, 2004–2016
| January 1, 2004–April 14, 2016 |
January 1, 2004–December 31, 2014 |
January 1, 2015–April 14, 2016 |
|||||
| Variable | Death Certificate | Fatal Encounters | Death Certificate | WGBH News | Death Certificate | The Counted | Washington Post |
| No. of cases | 84 | 84 | 69 | 68 | 15 | 15 | 14 |
| Age, y, mean (SD) | 34.6 (11.8) | 34.7 (11.8) | 33.7 (12.3) | 33.9 (12.3) | 38.7 (8.9) | 38.9 (8.9) | 37.8 (8.7) |
| Concordance with death certificate, Pearson r (95% CI) | NA | > 0.99 (0.99, 1.00) | NA | > 0.99 (0.99, 1.00) | NA | > 0.99 (0.99, 1.00) | > 0.99 (0.99, 1.00) |
| Missing, no. (%) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 1 (7.1) |
| Gender, no. (%)a | |||||||
| Men | 81 (96.4) | 81 (96.4) | 66 (95.7) | 65 (95.6) | 15 (100.0) | 15 (100.0) | 14 (100.0) |
| Women | 3 (3.6) | 3 (3.6) | 3 (4.3) | 3 (4.4) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Concordance with death certificate, % agreement; Cohen’s κ (95% CI)b | NA | 100; 1.00 (0.53, 1.00) | NA | 100; 1.00 (0.53, 1.00) | NA | 100c | 100c |
| Missing, no. (%) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Race/ethnicity, no. (%)a | |||||||
| Blackd | 26 (31.0) | 19 (32.8) | 22 (31.9) | 17 (29.8) | 4 (26.7) | 4 (26.7) | 4 (28.6) |
| Hispanic/Latino | 13 (15.5) | 10 (17.2) | 9 (13.0) | 7 (12.3) | 4 (26.7) | 4 (26.7) | 3 (21.4) |
| White | 45 (53.6) | 29 (50.0) | 38 (55.1) | 33 (57.9) | 7 (46.7) | 7 (46.7) | 7 (50.0) |
| Asian/Pacific Islander | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| American Indian Alaska Native | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Concordance with death certificate, % agreement; Cohen’s κ (95% CI)b | NA | 94.8; 0.92 (0.73, 1.00) | NA | 96.5; 0.94 (0.74, 1.00) | NA | 100; 1.00 (0.64, 1.00) | 100; 1.00 (0.62, 1.00) |
| Missing, no. (%) | 0 (0.0) | 26 (31.0)e | 0 (0.0) | 11 (16.2)e | 0 (0.0) | 0 (0.0) | 0 (0.0) |
Note. CI = confidence interval; NA = not applicable. Percentages might not add to 100 because of rounding.
Percent distribution based on observed values; percent of missing based on total population.
WGBH data included a “person of color” category (n = 3). When the death certificate indicated that the decedent was Cape Verdean, we considered person of color to be a concordant pair (n = 2). In one other case when the death certificate did not indicate Cape Verdean, we considered race to be missing in the WGBH data.
Cohen’s κ could not be computed because all cases were in the same category (i.e., men).
A total of 3 Cape Verdean decedents were coded as Black.
We failed to reject the null hypothesis of equal missingness rates by death certificate race/ethnicity for WGBH data (Fisher exact test, P = .72) and Fatal Encounters data (Fisher exact test, P = .45).
DISCUSSION
Our central finding was that death certificates corroborated all 84 Massachusetts legal intervention deaths reported in the 4 journalistic sources between 2004 and 2016 (16 media-reported cases did not meet the definition). In addition, these journalistic sources provided highly valid demographic data that were consistent with the gold standard of death certificate–reported values.
We also observed that deaths misclassified on the death certificates often had non-firearm-related mechanisms, or the medical examiner did not mention law enforcement on the death certificate. During the study period, 24% of the death certificates for cases reported by the media were misclassified (declining from 50% in 2004–2006 to 11.5% in 2013–2015), which was lower than the 42% misclassification identified in a published nonrandom selection of 16 states during 2005 to 2012.4 Misclassification and undercounts of legal intervention deaths affect multiple other US states.3,4
Study findings may not be generalizable to all US states, which have diverse systems for monitoring deaths (e.g., coroner-based systems). Also, because no deaths were reported for American Indian/Alaska Native or Asian/Pacific Islander persons over the study period, we could not draw conclusions about data for those groups.
PUBLIC HEALTH IMPLICATIONS
Previous research suggests that news media–based data sources document higher counts of legal intervention deaths compared with vital statistics.4 Health departments can use the methods described earlier in this article to quantify the extent of misclassification and address its causes. Media reports also may serve as a supplemental data source to identify legal intervention cases—to be used, for example, to characterize deaths in the National Violent Death Reporting System4 or in efforts to provide more accurate, timely data to the public.12 Finally, our results suggest that demographic data provided in media-based sources show promise for their use in epidemiological research on legal intervention deaths.
Improving monitoring of legal intervention is important to public health because it is a cause of health inequities with broad social implications, and health departments can serve as an official data source to inform policy interventions aimed at reducing police violence.12
HUMAN PARTICIPANT PROTECTION
Our study was deemed exempt by the Harvard T. H. Chan School of Public Health Human Subjects Committee (Protocol IRB16-0597) because it involved no living participants.
REFERENCES
- 1.Tran M. FBI chief: “unacceptable” that Guardian has better data on police violence. The Guardian. October 8, 2015. Available at: https://www.theguardian.com/us-news/2015/oct/08/fbi-chief-says-ridiculous-guardian-washington-post-better-information-police-shootings. Accessed December 5, 2016.
- 2.Banks D, Couzens L, Blanton C, Cribb D. Arrest-Related Deaths Program Assessment. March 2015. Available at: http://www.bjs.gov/content/pub/pdf/ardpatr.pdf. Accessed December 5, 2016.
- 3.Loftin C, McDowall D, Xie MA. Underreporting of homicides by police in the United States, 1976-2013. Homicide Stud. 2017;21(2):159–174. [Google Scholar]
- 4.Barber C, Azrael D, Cohen A et al. Homicides by police: comparing counts from the National Violent Death Reporting System, vital statistics, and supplementary homicide reports. Am J Public Health. 2016;106(5):922–927. doi: 10.2105/AJPH.2016.303074. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sorlie PD, Rogot E, Johnson NJ. Validity of demographic characteristics on the death certificate. Epidemiology. 1992;3(2):181–184. doi: 10.1097/00001648-199203000-00018. [DOI] [PubMed] [Google Scholar]
- 6.Arias E, Heron M, Hakes JK. The validity of race and Hispanic-origin reporting on death certificates in the United States: an update. Vital Health Stat 2. 2016;(172):1–21. [PubMed] [Google Scholar]
- 7.WGBH News. Map: 71 Police-involved deaths in Massachusetts from 2004-2014. December 16, 2014. Available at: http://projects.wgbhnews.org/police-involved-deaths. Accessed April 15, 2016.
- 8.Fatal Encounters Web site. Available at: http://fatalencounters.org. Accessed April 15, 2016.
- 9.The Counted: People killed by police in the US. The Guardian. Available at: http://www.theguardian.com/thecounted. Accessed April 15, 2016. [Google Scholar]
- 10.Fatal Force [online police shooting database] Washington Post. Available at: http://www.washingtonpost.com/policeshootings. Accessed April 15, 2016.
- 11.Massachusetts Department of Health and Human Services. Massachusetts Registry of Vital Records and Statistics: Genealogical Research. Available at: http://www.mass.gov/eohhs/gov/departments/dph/programs/admin/dmoa/vitals/genealogical-research.html. Accessed April 15, 2016.
- 12.Krieger N, Chen JT, Waterman PD, Kiang MV, Feldman J. Police killings and police deaths are public health data and can be counted. PLoS Med. 2015;12(12):e1001915. doi: 10.1371/journal.pmed.1001915. [DOI] [PMC free article] [PubMed] [Google Scholar]