Selection bias and misclassification in case–control studies conducted using the National Violent Death Reporting System

Abstract

Conducting case–control studies using the National Violent Death Reporting System (NVDRS) has the potential to introduce selection bias and misclassification through control selection. Some studies that use NVDRS compare groups of individuals who died by one mechanism, intent or circumstance, to individuals who died by another mechanism, intent or circumstance. For aetiological studies within NVDRS, the use of controls who had a different type of violent death has the potential to introduce selection bias, while relying on narrative summaries for exposure measurement may result in misclassification. We discuss these two methodological issues, and identify an unusual circumstance in which selection of live controls within NVDRS can be employed.

Case–control studies facilitate the study of possible aetiologies of rare outcomes that would be challenging to assess using prospective studies. However, the validity of a case-control study is threatened to the extent that the distribution of exposure among controls does not reflect that of the underlying population from which the cases arose (selection bias), or the ascertainment of exposure is not comparable between cases and controls (misclassification).¹

Case-control studies conducted using the National Violent Death Reporting System (NVDRS) are especially at risk for selection bias and misclassification through control selection within NVDRS, as NVDRS is a dataset which collects information only on incidents in which a violent death occurred. NVDRS is a state-based surveillance system, and enumerates homicides, suicides, legal intervention deaths, firearm-related deaths and some deaths with undetermined intent that may have been due to violence. NVDRS began in 2002 collecting data from six states; as of 2018, it included data from all 50 states as well as Puerto Rico and the District of Columbia. For each death, state-level abstractors gather data from law enforcement, coroner and medical examiners, toxicology reports, and death certificates to describe the incident circumstances, coding over 600 unique data elements (eg, suspect characteristics, injury mechanism and location of death) as well as writing narrative summaries of the medical and law enforcement reports.² It is a rich data source which has supported many descriptive studies, risk factor analyses and time trend analyses.³

Individual entries in NVDRS are limited to persons who have died, and data are deidentified at the state level before being collected at the national level. Linking with external data sources occurs only rarely and is quite time intensive.⁴ Many studies that utilise NVDRS compare groups of individuals who died by one mechanism, intent or circumstance, to individuals who died by another mechanism, intent or circumstance. For example, Choi et al compared youth who had committed suicide with a firearm to those who had died by suicide using other means, with the goal of identifying demographic and other characteristics (eg, mental health problems, criminal or legal problems) as possible predictors of suicide by means of a firearm.⁵ Comparisons of this sort are case–control studies, even if not explicitly specified as such, since they condition on the outcome (some feature of the death) and look retrospectively at prior exposures.

If the goal of a comparison of the type described above is to assess potential aetiological factors for a particular type of violent death, the use of controls who had a different type of violent death may introduce selection bias. Specifically, selection bias would be introduced when the prevalence of the exposure in the deceased controls does not reflect that of the population at risk from which the cases were derived.⁶ Therefore, associations observed (or the lack thereof) can be driven as much by risk or protective factors for the type(s) of death among controls as by as those for the type of death under study (ie, cases). Conceptually, this is similar to Berkson’s bias, which describes this phenomenon within the context of cases and controls hospitalised for different underlying conditions.^7,8 This issue has not often been addressed explicitly by the authors of studies conducted within the data collected by the NVDRS. Say, for example, we are interested in studying the termination of a teen dating relationship as a risk factor for teen suicide. We might consider defining ‘cases’ as suicides among teenagers who had no antecedent serious illness, and ‘controls’ as suicides in which such an illness was present. If teens who die by suicide following a serious illness tend not to have entered into a teen dating relationship because of their illness, then they would not represent the underlying population of youth with respect to the proportion who recently terminated such a relationship. Selection of these controls would lead to a spuriously large association between the termination of a dating relationship and the occurrence of suicide among teens (table 1).

Table 1.

2×2 tables for hypothetical case-control study conducted using NVDRS to assess risk factors for teen suicide

	Dead controls: youth suicide decedents with antecedent serious illness				Ideal controls: controls drawn from same underlying population as cases
			Cases	Controls			Cases	Controls
Cases: Youth suicide decedents with no antecedent serious illness	Recent end of	+	60	10	Recent end of	+	60	30
	relationship	−	40	90	relationship	−	40	70
	OR			=13.5	OR			=3.5

NVDRS, National Violent Death Reporting System.

However, there are occasional instances in which NVDRS can be used to select live controls, and we believe these are less likely to be plagued by selection bias. For example, when evaluating risk factors for suicide by the perpetrator of a homicide in an intimate partner homicide, investigators could use the homicide-suicide variable to identify intimate partner homicides in which the perpetrator lived compared with intimate partner homicides in which the perpetrator died by suicide. This approach, used by Banks et al, minimises concern for selection bias as both the cases and controls are drawn from the same underlying population of intimate partners who experience violence within their relationship.⁹ As NVDRS includes some standardised data collection fields for perpetrators of homicides, in addition to data collected that are specific to the victims, such an approach would be feasible to identify cases and controls and to ascertain characteristics of these individuals in a comparable manner.

In addition, investigators could evaluate individual, perpetrator and incident characteristics associated with the presence of corollary victims (eg, family or witnesses killed during an intimate partner homicide) versus similar incidents without corollary victims (eg, family or witnesses present but not killed during an intimate partner homicide). The narrative summaries of the medical and law enforcement reports included in NVDRS often include descriptions of family, friends and witnesses who did not die during the incident, as well as events and circumstances preceding the incident to understand the context of the death. As NVDRS does not routinely collect information on persons who are neither victims nor perpetrators, this requires a much more intensive approach to ascertaining control eligibility through narrative review as well as abstracting exposure information about controls. One could use some form of natural language processing or a comprehensive list of search terms to first identify narratives that may yield eligible controls, and then: (1) conduct a manual review to confirm eligibility before conducting data abstraction or (2) review all narratives for incidents identified as providing potential controls using NVDRS coded variables. These approaches have been used in NVDRS previously.^10,11 While most deaths are accompanied by narrative summaries, the amount of detail contained varies greatly between incidents (from a few words to multiple paragraphs). Thus, the completeness of exposure ascertainment would be limited by the accuracy and depth of detail recorded in the narratives. While level of detail and whether or not there is a description of the exposure of interest may vary considerably, the use of these narrative summaries to assess preceding events and circumstances in NVDRS studies is common.^12,13 However, the risk of exposure misclassification when using the narratives for exposure ascertainment is higher than when relying on NVDRS abstracted variables for both cases and controls.

In some instances, it may not be possible to identify dead or live controls within NVDRS who do not introduce selection bias. This can be illustrated with the earlier example of teen dating relationships and youth suicide. Selecting youth suicide decedents with antecedent illness as controls can clearly introduce selection bias; similarly, any subset of suicide decedents would likely introduce selection bias. Potential control alternatives could include friends or siblings of the youth suicide decedents, but it is unlikely that NVDRS abstractors would have had or included information on dating relationships of the siblings or friends of suicide decedents. Looking outside the pool of suicide decedents for potential controls would similarly introduce bias. Youth with non-suicide-related violent deaths (eg, homicide, legal intervention) who would be included in NVDRS would be unlikely to come from the same underlying population as youth suicide decedents. It is possible that identification of dead and live controls within NVDRS that reduce bias may not be possible when suicide decedents are the cases.

However, the use of NVDRS as a means of identifying cases for a case-control study does not preclude the use of controls from another source altogether. For example, a study by Boulifard and Pescosolido used the American Community Survey to compare population-based controls to suicide decedents in NVDRS.¹⁴ However, in such studies, the potential aetiological factors may not have been ascertained in the same way for NVDRS cases and controls from the alternative source,^15,16 which could lead to exposure misclassification. Whatever the sampling frame used in selecting controls for a study that employs cases derived from NVDRS, we recommend an acknowledgement be made of the specific types of bias that the method of control selection may have introduced.

What is already known on the subject

• The National Violent Death Reporting System (NVDRS) is widely used for injury prevention research.

• Case-control studies using dead cases and controls from NVDRS are often subject to selection bias or misclassification.

What this study adds

• Occasionally, there are research questions for which live control selection is possible within National Violent Death Reporting System (NVDRS).

• Investigators who use the NVDRS for aetiological studies would do well to consider the possibility of selection bias and misclassification when interpreting and describing their results.