Methodological limitations in the measurement and statistical modeling of violence among adults with mental illness

Abstract

Objectives

Methodological limitations of extant research hinder the development of effective violence risk screening, assessment, and management strategies for adults with mental illness. This study quantifies the effects of three common limitations: (a) insensitive measurement of violence that results in violence classification with high levels of information bias, (b) use of cross‐sectional data, and (c) use of data lacking spatiotemporal contiguity.

Methods

We utilize secondary data (N = 3,000 participants; N = 10,017 observations) and parametric and nonparametric bootstrap simulation methodologies.

Results

Not utilizing self‐reported violence data increases information bias. Furthermore, cross‐sectional data that exclude self‐reported violence produce biased associations between substance use and psychiatric symptoms and violence. Associations between baseline variables and subsequent violence attenuate over longer time lags and, when paired with high levels of violence information bias, result in fewer significant effects than should be present. Moreover, the true direction of the simulated relationship of some significant effects is reversed.

Conclusions

Our findings suggest that the validity of conclusions from some extant research on violence among adults with mental illness should be questioned. Efforts are needed to improve both the measurement of violence, through inclusion of self‐report, and the statistical modeling of violence, using lagged rather than nonlagged models with improved spatiotemporal contiguity.

1. INTRODUCTION

Despite equivocal evidence linking mental illness to serious violence (Knoll & Annas, 2015; Large, Smith, Swinson, Shaw, & Nielssen, 2008; Nielssen, Westmore, Large, & Hayes, 2007), recent high‐profile incidents, including mass shootings, have resulted in increased scrutiny on clinicians' ability to identify and target violence‐related factors among adults with mental illness (Metzl & MacLeish, 2015; Swanson, 2008). Similar scrutiny exists regarding the degree to which current empirical evidence can inform violence risk screening, assessment, and management strategies (Fazel, Singh, Doll, & Grann, 2012). The implications are clear: Valid, reliable, and generalizable research is better able to inform effective intervention development and implementation than research with limited validity, reliability, and generalizability (Douglas, Otto, Desmarais, & Borum, 2012). To that end, we contend that some extant research hampers effective violence risk screening, assessment, and management efforts because of three methodological limitations: (a) insensitive measurement of violence, (b) use of nonlagged or cross‐sectional data in regression models, and (c) a lack of spatiotemporal contiguity between leading indicators and subsequent outcomes.

1.1. Measurement of violence

Self‐report is a valid and reliable measure for collecting violence data from adults with mental illness (Crisanti, Laygo, & Junginger, 2003; Harris, Oakley, & Picchioni, 2013; Huizinga & Elliott, 1986; Van Dorn et al., 2010). Other data sources, such as collateral informant reports or treatment and arrest records, are sometimes used to augment self‐reported data on violence (Steadman et al., 1998; Swanson, Swartz, & Elbogen, 2004). When combined, the use of multiple data sources has been shown to increase violence detection over self‐report alone, although self‐report captures the large majority of violence perpetration (Mulvey, Shaw, & Lidz, 1994; Steadman et al., 1998).

Nevertheless, some research relies on other data sources to the exclusion of self‐report. As a recent example, Coid, Kallis, Doyle, Shaw, and Ullrich (2015) measured violence through treatment and arrest records and clinician collateral reports but did not collect patient self‐report when examining community‐based violence. Other examples have used either arrest and medical records' abstraction data (Douglas, Ogloff, Nicholls, & Grant, 1999; Nicholls, Ogloff, & Douglas, 2004) or arrest records alone (De Vries Robbé, de Vogel, Douglas, & Nijman, 2015) to obtain information on violence perpetration in community or inpatient settings. However, these sources miss instances of violence that do not rise to the level of an official report, in the use of arrest or medical records' abstraction data, or when the collateral informant did not witness the event. Subsequently, excluding self‐report likely underestimates violence perpetration and biases multivariable regression results (King & Zeng, 2001). More specifically, both the magnitude and direction of the misspecification of nonviolent and violent cases shift distributions away from the “true” relationship between cause and effect and misguide clinical risk identification and management efforts among adults with mental illness.

1.2. Lagged data and regression models

Effective risk screening, assessment, and management strategies should be informed by evaluation of how changes in independent variables—through exacerbation of risk factors or enhancement of protective factors—are associated with changes in outcomes (i.e., violence; Douglas & Skeem, 2005; Skeem & Monahan, 2011). In multivariable regression models, this information is best obtained from lagged data that assess causes and effects, or leading indicators and changes, over time (Van Dorn et al., 2017; Van Dorn, Volavka, & Johnson, 2012). Unfortunately, much research on mental illness and violence has used cross‐sectional data, which are “essentially useless” for establishing the direction of causal effects (VanderWeele, Jackson, & Li, 2016, p. 1465) or identifying clinically relevant change (Johnson, Desmarais, Tueller, et al., 2016). Cross‐sectional designs are unable to distinguish if the clinically relevant risk or protective factor occurred prior to the violence perpetration, which is a necessary (but not sufficient) criteria for causality (Kraemer et al., 1997; Van Dorn et al., 2012), nor can these designs establish how close in time they occurred.

There have been a handful of efforts to address temporality by lagging clinically relevant independent and dependent variables (Van Dorn et al., 2012; Van Dorn et al., 2017; Skeem et al., 2006; Odgers et al., 2009; Swanson et al., 2008). Such lagged models can then, depending on other features of the study design and measurement, be appropriately discussed with respect to either causality or prediction. However, even when lagged models have been examined, the language used regarding statistical prediction and causality is sometimes unclear or simply incorrect (for a review of one such example, see Van Dorn et al., 2012).

1.3. Spatiotemporal contiguity

A deficiency across many fields of research is the lack of longitudinal designs with a high enough measurement frequency to appropriately address spatiotemporal contiguity—that is, for example, both the measurement spatial gap, from the leading indicator to the outcome, and the temporal delay between the clinical change in some symptom and a change in the likelihood of violent behavior. Specific to research on violent outcomes and mental illness and violence risk assessment and management practice, the optimal spatial gap for the measurement of leading indicators and outcome is unknown, as are average temporal delays between relevant symptoms and subsequent violence (Douglas & Skeem, 2005). Indeed, it is possible that different leading indicators have different optimal lags or spatial gaps and temporal delays. Given that clinically relevant factors are dynamic and may vary over short periods of time (Appelbaum, Robbins, & Vesselinov, 2004), data with 3‐, 6‐, or 12‐month lags seem less desirable than data with shorter lags (Sariaslan, Lichtenstein, Larsson, & Fazel, 2016). Still, the literature is replete with studies using data with long lags. As an example, one study found a relationship between violent victimization and a functional outcome score (Hodgins, Lincoln, & Mak, 2009), but both the violent event and functional outcome were assessed concurrently, and lagged clinical symptoms were assessed 1 year prior to the outcome. Thus, findings only demonstrated that the violent event and one's functional outcome covaried and that symptoms assessed 1 year prior had little to no bearing on current functioning. Other examples examined an indicator of lifetime mental illness in relation to recent violence perpetration (Elbogen & Johnson, 2009), and clinical symptoms assessed 6 months or up to a year or more before violence were assessed (Coid et al., 2015; Link, Stueve, & Phelan, 1998; Monahan et al., 2001; Roaldset & Bjørkly, 2015; Sadeh, Binder, & McNiel, 2014; Stompe, Ortwein‐Swoboda, & Schanda, 2004; Swanson, Borum, Swartz, & Monahan, 1996; Swanson et al., 1997; Swanson & Van Dorn, 2010). There are numerous etiologies of—and causal pathways to—violence, and targets of clinical interventions and violence risk management strategies may be misdirected, or the effects of such efforts may be missed altogether with long data lags and unrelated leading indicators.

1.4. Present study

The present study seeks to quantify the effects of insensitive measurement of violence, differences between nonlagged and lagged models, and different time lags between leading indicators and outcomes. We utilize existing data from three studies (Lieberman et al., 2005; Steadman et al., 1998; Swanson et al., 2004) to determine rates of violence misclassification, or information bias (cf. Althubaiti, 2016) that results from excluding self‐reported data (Research Aim 1). Via bivariate logistic regression analysis, we examine how conclusions regarding associations of independent variables, specifically substance use and psychiatric symptoms, with violence are affected by nonlagged and lagged specifications (Research Aim 2). Then, using multiple parametric and nonparametric bootstrap simulation methodologies, we assess the effects of different data time lags (Research Aim 3).

To illustrate the issues under investigation in Research Aims 2 and 3, Figure 1 displays four potential specifications between a purported risk factor and violence. Figure 1a is illustrated using the autoregressive cross‐lagged model (Van Dorn et al., 2017), in which knowledge of a person's score on a leading indicator provides information about subsequent scores on the same (autoregressive effect) or other variables (cross‐lagged effect). Figure 1a shows both the ideal longitudinal data specification and the less ideal, albeit more typical, specification. Specifically, associations presented in gray and black show the effect (e.g., r₂v₃) of a leading indicator (e.g., risk₂) that has close precedent temporality to a violent act (e.g., violence₃). However, typical longitudinal sampling designs only allow for the estimation of effects at longer lags (e.g., r₁v₃).

Figure 1.

Statistical models of the effect of risk on violence

Figure 1b takes the specification of Figure 1a and adds cross‐sectional effects (i.e., r₁v₁, r₂v₂, and r₃v₃) not capable of establishing causality. Indeed, in the absence of evidence that the purported risk factor preceded the violent act (Kraemer et al., 1997), it is possible that the violent act occurred first (i.e., v₁r₁, v₂r₂, and v₃r₃). The multilevel (or mixed effects) version of the repeated cross‐sectional model of Figure 1b is given in Figure 1c, again using a dashed line to denote the ambiguity present when modeling cross‐sectional data. Finally, Figure 1d includes the one effect estimable in the multilevel lagged model from Figure 1a (i.e., r₁v₃). The variance parameters—V(v₃r₃) and V(r₃v₃) and V(v₁r₃) in Figure 1c,d, respectively—show the random effect of a given risk factor on violence, though these are estimated at the cost of failing to control for r₁r₃ and v₁v₃. All models can be generalized such that Times 1 and 3 represent any two observed time points and 2 represents any unobserved time point.

2. METHODS

The study protocol was approved by institutional review boards from RTI International and North Carolina State University.

2.1. Data sources

Data were integrated from the (a) Clinical Antipsychotic Trials of Intervention Effectiveness Study (Lieberman et al., 2005), (b) MacArthur Mental Disorder and Violence Risk Study (Steadman et al., 1998), and (c) Schizophrenia Care and Assessment Program (Swanson et al., 2004). Studies included broad inclusion and minimal exclusion criteria and enrolled a range of participants, all of whom gave written informed consent. (See Van Dorn et al., 2017, for more detail regarding the statistical integration of the five datasets used for the parent project and from which the three datasets with multiinformant violence data were drawn for this study.)

Participants in the Clinical Antipsychotic Trials of Intervention Effectiveness Study (n = 1,460) were recruited from 57 sites across the United States. Inclusion criteria were (a) 18–65 years of age, (b) schizophrenia, and (c) ability to take oral antipsychotics. Data were collected between 2001 and 2004. Violence was assessed at baseline and every 6 months for up to 18 months. Substance use and psychiatric symptoms were assessed at baseline, 1, 3, 6, 9, 12, 15, and 18 months.

Participants in the MacArthur Mental Disorder and Violence Risk Study (n = 1,136) were recruited from three sites in Pennsylvania, Missouri, and Massachusetts. Inclusion criteria were (a) English‐speaking Caucasian, African American, or Hispanic patients; (b) 18–40 years of age; and (c) schizophrenia spectrum, depression, mania, brief reactive psychosis, delusional disorder, “other” psychotic disorder, substance abuse/dependence, or personality disorder. Data were collected between 1992 and 1995. Assessments were conducted at baseline and every 10 weeks for a total of five follow‐up assessments.

Participants in the Schizophrenia Care and Assessment Program (n = 404) were recruited from treatment facilities across North Carolina. Inclusion criteria were (a) 18–65 years of age, (b) schizophrenia, and (c) current service use. Data were collected between 1997 and 2002. Assessments were conducted at baseline and every 6 months for 36 months.

2.2. Measures/variables

Self‐reported violence was measured using the MacArthur Community Violence Screening Instrument (MCVSI; Cartwright, Desmarais, Grimm, Meade, & Van Dorn, in press; Desmarais et al., 2014; Steadman et al., 1998). The MCVSI includes eight behaviorally based self‐reported questions that assess (a) pushing, grabbing, or shoving; (b) kicking, biting, or choking; (c) slapping; (d) throwing an object; (e) hitting with a fist or object; (f) sexual assault; (g) threatening with a weapon in hand; and (h) using a weapon. Self‐reported violence was defined as a positive response to at least one item. Non‐self‐reported violence was measured via abstraction of official arrest records and inpatient/outpatient medical records (Swanson et al., 2004), combined treatment records and collateral informant reports (Steadman et al., 1998), and collateral informant report (Swanson et al., 2006). Non‐self‐reported violence was defined as the presence of violence noted in at least one of these other sources. A total of 10,017 violent/nonviolent observations were included across all sources of information.

The CAGE (Mayfield, McLeod, & Hall, 1974), the Alcohol and Drug Use Scales (Drake et al., 1990), urine drug screens, self‐report, and the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (First, Spitzer, Gibbon, & Williams, 1996) were used to indicate abstinence, nonproblematic, and problematic alcohol and drug use (Van Dorn et al., 2017). The Brief Psychiatric Rating Scale (Overall, 1974) and the Positive and Negative Syndrome Scale (Kay, Fiszbein, & Opfer, 1987) were used to assess psychiatric symptoms via affective, positive, negative, and disorganized cognitive processing factors (Tueller et al., 2017; Van Dorn et al., 2016).

2.3. Statistical analyses

2.3.1. Violence perpetration information bias

We examined the sensitivity, specificity, positive predictive value, negative predictive value, Kappa, false positive rate, false negative rate, accuracy, and misclassification rate associated with excluding self‐report and instead relying on non‐self‐reported violence (Research Aim 1).

2.3.2. Nonlagged versus lagged specifications

We estimated the effects of substance use and psychiatric symptoms on violence perpetration in both nonlagged and lagged models (Research Aim 2). Bivariate logistic regression was used to estimate odds ratios (ORs). Violence was specified with and without information bias, and an indicator of misclassification was included as an additional outcome to show whether baseline factors were related to the likelihood of misclassification.

2.3.3. Time lags

We examined a linear degradation of the relationship between baseline factors and follow‐up violence at 2, 4, 10, 17, 26, and 52 weeks (Research Aim 3). The initial effects of a given baseline variable on violence over a 1‐week period were specified as a small risk factor (i.e., OR = 1.5), a small protective factor (i.e., OR = 0.67), and null effect (i.e., OR = 1.00). Because only the baseline variable was used, attenuation of the OR is due to changes in violence resulting from the hypothetical accumulation of unobserved factors. At each time point, data were randomly misclassified using the misclassification rate reported in the next section. This was repeated 1,000 times, resulting in a distribution of possible consequences of misclassification.

3. RESULTS

3.1. Research Aim 1: Violence perpetration information bias

Reliance on non‐self‐reported violence (i.e., collateral reports, treatment records, and arrest records) failed to capture most violent incidents detected by self‐report (Figure 2).

Figure 2.

Prevalence of positive violent observations by source

Using self‐report as the criterion, there were 1,484 observations of violence and 8,533 observations of no violence. Among violent incidents, 36.9% were classified correctly (sensitivity); among cases with no violence, 95.0% were classified correctly (specificity). Of those observations classified as violent by non‐self‐reported measures, 56.1% were correct (positive predictive value); among those classified as not violent, 89.6% were correct (negative predictive value). Excluding self‐report missed 63.1% of all violent incidents (false negative rate), and non‐self‐reported measures captured only 5% of violence that was not self‐reported (false positive rate). The Kappa value of 0.37 is indicative of poor levels of agreement across measures. Of all observations, 86.4% were classified correctly, driven in large part by between‐source agreement on nonviolent observations; conversely, there was a misclassification rate of 13.6%. The effects of this misclassification rate are explored in the next two sections.

3.2. Research Aim 2: Nonlagged versus lagged specifications

Bivariate regression results comparing nonlagged versus lagged effects in the presence of violence information bias are presented in Table 1. Both nonlagged and lagged effects of baseline substance use on violence without information bias were significant, and lagged effects were larger. Information bias attenuated nonlagged, but not lagged, effects. Specifically, baseline substance use was significantly associated with baseline, but not 6‐month, information bias. Baseline psychiatric symptoms were generally not significantly associated with baseline violence, but baseline symptoms were significantly associated with 6‐month violence (without and with violence information bias). Psychiatric symptoms were not associated with baseline or 6‐month information bias.

Table 1.

Cross‐sectional and lagged effects of risk factors on violence status, misclassified violence status, and a binary indicator of misclassification

Baseline risk	Baseline violence (cross‐sectional effect)			6‐month violence (lagged effect)
	Any reported violence	Misclassified violence (excluding self‐report)	Misclassification indicator	Any reported violence	Misclassified violence (excluding self‐report)	Misclassification indicator
Substance use
Alcohol	1.46*	1.16	1.95*	1.80*	1.87*	1.29
Drugs	1.77*	1.35	2.29*	1.89*	1.91*	1.19
Psychiatric symptoms
Affective	0.98	1.03	0.91	1.38*	1.55*	0.92
Positive	1.19	1.12	1.27	0.74*	0.70*	0.95
Negative	0.88	0.78*	1.21	0.69*	0.63*	1.01
Disorganized cognitive processing	0.93	0.88	1.07	0.62*	0.57*	0.95

Note. The misclassification rate of 13.6% was determined from existing data where self‐reported violence perpetration data were not used but instead non‐self‐report measures were used.

^* p < 0.05, and cell entries are odds ratios.

3.3. Research Aim 3: Time lags

Results of the simulation study examining time lags are given in Figure 3, where the protective factor effect is presented on the left, the null effect in the middle, and the risk factor on the right. The dotted line shows the degradation of the effect of a baseline variable on follow‐up violence up to 52 weeks in the absence of violence information bias; time points denoted with an asterisk indicate that the baseline variable was significant at that follow‐up length. The solid line shows the average effect under the 13.6% misclassification rate identified above, and the superimposed boxplots show the distribution of misclassification (median, first and third quantiles as the box ends, 95% confidence interval at the whisker ends, and outliers). When there is an association between the baseline effect and violence information bias, results will systematically vary from the average line.

Figure 3.

Simulated effect of increasing time lags under nonmisclassified and misclassified violence status for a protective factor, a null effect, and a risk factor. For the misclassified data, percentages are the percent of replications in the simulation study for which p < 0.05 when the effect was positive (odds ratio [OR] > 1) whereas percentages within parentheses are the percent of replications for which p < 0.05 when the effect was negative (OR < 1)

The true simulated effect of the baseline protective factor remained significant through Week 10 (Week 2, p = 0.03; Week 4, p = 0.03; Week 10, p = 0.04; Week 17, p = 0.08; Week 26, p = 0.13; Week 52, p = 0.48), whereas the true simulated effect of the risk factor remained significant through Week 17 (Week 2, p = 0.02; Week 4, p = 0.02; Week 10, p = 0.03; Week 17, p = 0.04; Week 26, p = 0.07; Week 52, p = 0.30). However, on average, the protective and risk factor effects associated with violence information bias were not significant. That is, only 12.4%, 13.8%, and 14.4% of the 1,000 misclassification bootstraps at Weeks 2, 4, and 10, respectively, indicated an OR < 1.0 (p < 0.05). For the risk factor effect, at Weeks 2, 4, 10, and 17, only 14.5%, 12.7%, 10.4%, and 9.1% of the 1,000 misclassification bootstraps indicated an OR > 1.0 (p < 0.05). Finally, at no time did the simulated true null effect approach significance (all ps = 0.97). Between 1.0% and 2.3% of misclassified simulated results indicated significant effects that were either positive or negative when the effect should have been null.

4. DISCUSSION

This paper quantified the effects of three common methodological limitations in research on mental illness and violence: insensitive measurement of violence that results in violence information bias (Research Aim 1), nonlagged or cross‐sectional data used in regression models (Research Aim 2), and long lags between leading indicators and subsequent outcomes (Research Aim 3). Taken together, our findings suggest that the conclusions of some extant work on risk and protective factors for violence among adults with mental illness should be viewed skeptically, particularly when used to develop, validate, and implement valid risk screening, assessment, and management strategies. We summarize significant findings below.

Our findings suggest that reliance on non‐self‐reported measures of violence (e.g., arrest records) to the exclusion of self‐report results in high rates of violence misclassification or information bias, as evidenced most acutely by an unacceptably high false negative rate. The current study represents an advance over prior research that has examined agreement between different sources of information related to violence perpetration. Although such prior work (Mulvey et al., 1994; Steadman et al., 1998) has examined intersource agreement, we are not aware of any study that has examined how a lack of agreement and a high false negative rate result in biased effects across different statistical models and data specifications. That is, nonlagged or cross‐sectional models that exclude self‐report produce biased estimates of associations between purported risk and protective risk factors and violence. Moreover, these models are unable to show how changes in independent variables are associated with changes in outcomes over time. Finally, statistical associations between baseline variables and violence attenuate over longer time lags and, when paired with violence information bias, also result in fewer observed significant effects than should be present, in addition to observed significant effects that are reversed from the true direction of the relationship.

4.1. Implications for research, clinical care, and policy

Violence perpetration by adults with mental illness is complex and multidetermined and, consequently, difficult to predict (Swanson et al., 2006). As demonstrated herein, research that misclassifies violence and then utilizes nonlagged models or has long lags between predictors and outcomes results in findings with limited validity, reliability, and generalizability and, in the end, would appear to hamper clinicians' abilities to effectively fulfill their legal and ethical obligations related to violence risk screening, assessment, and management. Consequently, these methodological issues increase the likelihood of introducing unnecessary harms into the clinical dynamic, both for the client subjected to the violence risk screen/assessment and for the treating clinician. However, we strongly disagree with the notion that there is little clinicians can do to reduce violence, even if they were better at predicting it (Swanson, 2008). We believe that this bleak outlook is misguided for two primary reasons.

First, because of the methodological issues reviewed herein, extant research has likely contributed to incorrect conclusions regarding both the performance of screening and assessment instruments and the effectiveness of clinical interventions in reducing violence risk. Regarding screening and assessment instruments, a recent analysis (Cartwright, Desmarais, Johnson, & Van Dorn, 2018) provides an example of how one proposed screening instrument, developed and validated on secondary registry data (Singh, Grann, Lichtenstein, Långström, & Fazel, 2012), differs in performance when evaluated against self‐reported violence data. Regarding attempts to identify reductions in violence risk via psychosocial or psychopharmacological interventions, reductions in violence will not occur if clinical interventions target misidentified violence risk and protective factors that are not actually related to future violence. Indeed, research demonstrates that when interventions target empirically supported factors relevant to a patient's violence risk, there are subsequent reductions in violence (Singh et al., 2014). Care must be given in the provision of either compulsory or voluntary violence risk assessment and management efforts that target factors associated with an individual patient's risk of violence, and the targeting of factors should be based on valid research. To do otherwise ignores the potential for harms (Gaynes et al., 2017) associated with screening, assessment, and treatment designed to identify and reduce violence risk among adults with mental illness.

Second, for too long research and clinical efforts have focused on screening and assessing violence risk to the neglect of risk management. Moreover, violence risk screening and assessment are viewed by many clinicians as a one‐off endeavor with limited relevance to day‐to‐day practice. Such a view may be attributable to a confluence of issues, including clinician attitudes, (lack of) institutional support, and policy (Levin, Nilsen, Bendtsen, & Bulow, 2016), as well the limited capacity of many violence risk screening and assessment instruments (Cartwright et al., 2018; Large & Nielssen, 2017) to provide information that enhances clinicians' ability “to respond to changing circumstances and a changing clinical picture” (Buchanan, Binder, Norko, & Swartz, 2012, p. 8). As noted earlier, many clinically relevant factors, such as mood, may change dramatically over short periods of time (Appelbaum et al., 2004; Van Dorn et al., 2017)—and these changes cannot be captured in cross‐sectional designs or lagged designs with poor spatiotemporal contiguity. Violence risk screening and assessment and subsequent management are equally important aspects of clinical care (Haque, Cree, Webster, & Hasnie, 2008) that necessitate careful consideration of the measurement, data structure, and analytic approaches used to assess factors associated with violence perpetrated by adults with mental illness.

Though the presence of the three methodological limitations we examined are not unique to Coid and colleagues' work, and our intent is not solely to criticize or gainsay this groups' empirical approach and research decisions (cf. Coid et al., 2013; Ullrich, Keers, & Coid, 2013), points raised by these authors merit some discussion. In support of their cross‐sectional design, which they inappropriately identified as causal, Coid et al. (2015) state that “The risk factor may … no longer be present in a subsequent time window when violence is measured … [and] significant associations may consequentially be missed” (p. 2). However, it should be clear that cross‐sectional models are not a solution to a lack of spatiotemporal contiguity; research designs should observe hypothesized leading indicators and subsequent violence in closer temporal proximity. As results indicate, it is not that lagged effects are suboptimal in and of themselves but rather that the magnitude of the lag is not ideal for current measurement protocols. Specifically, simulation results (see Figure 3) show that the likelihood of observing any significant association between baseline protective or risk effects and subsequent violence beyond 10 or 17 weeks, respectively, is highly unlikely, even when violence is not misclassified. When violence is misclassified, significant effects are only found in approximately 1% of bootstrapped regression models through 17 weeks and even less in models with longer time lags. Expecting to identify valid, reliable, and clinically actionable leading indicator and subsequent violence perpetration relationships at 6 or 12 months is simply unrealistic, if the focus is truly on dynamic factors. As with the misinterpretation of cross‐sectional estimates, basing outcomes—and clinical recommendations—on data lacking spatiotemporal contiguity deleteriously affects violence risk management intervention development and implementation and introduces the potential of unnecessary harms to the client being assessed and to the client–clinician relationship.

An additional threat to research utilizing cross‐sectional designs is the fact that violence status may change over time (cf. from violence₁ to violence₃ in Figure 1a) in the absence of change in a measured risk factor (cf. from risk₁ to risk₂ in Figure 1a). For example, an individual may perpetrate violence at Time 1 and not at Time 2 but evince problematic substance use at Both Time 1 and Time 2. If risk₁ causes violence₃ and the risk indicator has not changed appreciably from risk₁ to risk₂, the observed null relationship from r₁v₃ may be due to the accumulation of unobserved risk and unobserved protective factors, or different opportunity structures for violence perpetration over time. Specifically, unobserved risk factors will increase violence risk for reasons other than the measured risk factor, whereas unobserved protective factors will reduce violence risk despite the presence of the measured risk factor. Finally, we cannot disentangle whether the difference between cross‐sectional and lagged effects is due to (a) the risk factor no longer being present (i.e., substantial changes from risk₁ to risk₂) or (b) the accumulation of unobserved risk and protective factors under no change in the measured risk factor (i.e., inconsequential changes from risk₁ to risk₂). (See Appendix S1 for a detailed overview of increasing and attenuating OR patterns related to nonlagged and lagged data specifications based on the presence or absence or unmeasured nature of risk factors and protective factors.)

4.2. Limitations

This study is not without limitations. First, we focused only on two risk factors—substance use and psychiatric symptoms—in our analysis of primary data. (In our simulation analysis, we did, however, examine the effects of both a hypothetical protective and risk factor, in addition to a null effect.) Although substance use and psychiatric symptoms are arguably two of the most analyzed and clinically relevant variables related to interventions for adults with mental illness (Van Dorn et al., 2017), other variables, such as medication adherence, social support, or attitudes towards treatment, could have been examined. Second, our quantification of the effects of three prominent limitations associated with extant research only applies to the assessment and management of violence risk for patients in clinical care. Adults with mental illness who do not participate in a community‐based system of care or drop out of care are obviously unable to have their violence risk screened, assessed, and managed in an ongoing manner. Third, our use of self‐report as the criterion measure in examining violence information bias is done with recognition that there is no “gold standard” for measuring violence. Indeed, additional measures may increase detection over self‐report alone (and the instances in which that is the case are noted in Figure 2). However, self‐report is singular in its ability to capture the majority of incidents of violence. Fourth, and finally, by focusing on a range of violent behaviors, as is done with the MCVSI, we are making the implicit argument that all violence matters in the context of clinical care. Registry‐based, and other secondary, administrative sources of violence data do not make this assumption, and thus research that relies on non‐self‐reported violence data is limited to violent crimes such as homicide, assault, robbery, and other criminal conduct likely to be reported to law enforcement or to come to the attention of clinicians or other inpatient/outpatient service providers. However, these violent crimes are a small minority of violence perpetrated by adults with mental illness (Desmarais et al., 2014). Additionally, research that focuses solely on violent outcomes based on non‐self‐reported data will, as shown in Figure 2, necessarily use data with lower violence base rates than data that include self‐reports of violence. Importantly, although data with increased base rates of violence do not produce larger effects, increases in the base rates of outcome variables will affect the OR, which has an interpretation that is base rate sensitive (cf. Persoskie & Ferrer, 2017). This observation underscores the argument that researchers should use the most accurate measures of violence, which includes, at a minimum, self‐report. This decreases the likelihood of analyzing misclassified base rates, which in turn results in more accurate leading indicator estimates (e.g., ORs) that can then be used to inform clinical care and mental health policies through valid violence risk screening, assessment, and management practices.

5. CONCLUSIONS

Quantifying the effects of violence misspecification, nonlagged versus lagged estimates, and different time lags, as we have done, should inform the development, validation, and implementation of violence risk screening and assessment instruments, in addition to the interpretation of existing research on mental illness and violence. Specifically, results indicate that the conclusions of studies that do not include self‐report, utilize cross‐sectional models for evaluating associations between clinically relevant variables and violence, and/or have data lags greater than 10 weeks for protective factors or 17 weeks for risk factors are likely limited in their validity. Thus, efforts are needed to improve the measurement of violence, through inclusion of self‐report, and the statistical modeling of violence, using lagged rather than nonlagged models with improved spatiotemporal contiguity, in research on violence risk among adults with mental illness. Still, other limitations remain, including the lack of rigorous research examining the effectiveness of violence risk screening and assessment instruments and management strategies in reducing violence among adults with mental illness (Desmarais, 2017). Only a handful of studies have examined this issue and have failed to produce uniformly positive results (Abderhalden et al., 2008; Kling, Yassi, Smailes, Lovato, & Koehoorn, 2011; Needham et al., 2004; Troquete et al., 2013; van de Sande et al., 2011). This remains a critical next step in the field.

Challenges to methodological or conceptual issues in violence risk assessment research are not new (Buchanan, 2008; Buchanan, 2014; Hart, Michie, & Cooke, 2007; Imrey & Dawid, 2015; Large & Singh, 2014; Mossman, 2014; Wand & Large, 2013). Our hope is that the current empirical examination of three limitations found frequently in the extant research will spur serious consideration about ways to improve the science and practice of clinical violence risk screening, assessment, and management.

DECLARATION OF INTEREST STATEMENT

The authors declare that they have no conflict of interest.

Supporting information

Figure S1. Transition plot, no misclassification. OR = Odds Ratio; No Alc = no alcohol misuse/abuse; Yes Alc = alcohol misuse/abuse; NonVio = non‐violent; N is the cell size in the transition table.

Data S1. Supporting information

Johnson KL, Desmarais SL, Tueller SJ, Van Dorn RA. Methodological limitations in the measurement and statistical modeling of violence among adults with mental illness. Int J Methods Psychiatr Res. 2019;28:e1776 10.1002/mpr.1776