External Validity of Studies on Aggressive Behavior in Patients with Schizophrenia: Systematic Review

Abstract

Studies on violence in schizophrenia use two different approaches: use of epidemiological data, and clinical studies recording direct patient data after gaining informed consent. With regard to informed consent requiring agreement and cooperation, the question arises as to what extent participants represent patients with schizophrenia and violent behaviour (external validity). We conducted a systematic literature research. In most of the studies, aggression or violence, respectively, were poorly defined. Only 5 (15.2%) studies used a cut-off score on an aggression scale. Only 6 studies (18.2%) reported the number of patients who refused to participate, and 16 (48.5%) reported the number of drop-outs. Only 3 studies (9.1%) reported a systematic comparison of participants and non-participants. We found that data which allow for the assessment of representativeness of the investigated samples are poorly reported. For most studies, doubts regarding external validity seem justified and generalisability is questionable due to possible selection bias.

INTRODUCTION

There is good evidence from epidemiological studies that the diagnosis of schizophrenia is associated with an increased risk of violence in the community [1], which persists with increasing age [2], is found in both males and females [3] and is strongly associated with co-morbid substance abuse [4, 5]. This poses a serious burden for relatives [6], staff, society as a whole and, last but not least, patients with psychoses themselves due to the subsequent stigmatisation [7]. Thus there is a considerable interest in conducting clinical studies on the origin and therapy of violent behaviour in these patients. However, clinical studies require written informed consent due to ethical reasons. In hospital settings, violent behaviour of patients with psychoses is associated with involuntary treatment [8], lack of insight into their illness and treatment [9, 10], cognitive distortion [11], disorganisation [12] and positive symptoms [13]. According to the nature of the disorder and the severely disturbed behaviour, violent psychotic patients are mostly uncooperative and distrustful, and lack the ability to understand and appropriately process new information. Therefore, these patients are typically neither willing nor able to give informed consent to study participation and as such it is obviously a significant challenge to conduct clinical studies with patients possessing these characteristics. Patients difficult to treat, e.g. with antisocial behaviour, show higher refusal rates than patients without antisocial behaviour [14]. Epidemiological studies find significantly divergent prevalence rates of aggression depending on how many sources of information the authors relied on [15] indicating that one single data source may cause bias. This indicates that studies on aggression in schizophrenia have to take into account risks of selection bias and underestimation of prevalence rates. More than a decade ago, Volavka and Citrome highlighted methodological concerns in research on atypical antipsychotics in the treatment of persistently aggressive psychotic patients, including expectations of doctors and patients in non-randomised trials, influence of environmental factors, use of proxy measures as outcomes, insufficient definition of aggression, consent selection bias, inappropriate observation periods, and unrevealed diagnostic confounders such as substance abuse [16].

Nevertheless, numerous randomised controlled trials on the treatment of aggressive patients with schizophrenia and meta-analyses of these trials have been conducted [17] and recommendations in treatment guidelines are based on these results [18, 19]. This poses the question of whether or not patients included in these studies are representative of those violent patients who are described in epidemiological studies and in descriptive studies of in-patient violence, i.e., whether these studies have sufficient external validity to allow for generalisability of the obtained results.

We conducted a systematic literature review on clinical studies of violent patients with schizophrenia. The primary objective was to analyse whether the published data provides information about a reference population and representativeness of the study sample. As a secondary objective, we wanted to obtain information on the average proportion of patients who are reluctant to participate or drop out after inclusion into the studies. This is important for the planning of future studies on such difficult-to-recruit populations.

METHODOLOGY

Search Strategy

We conducted a systematic search of relevant literature in the electronic databases PubMed (1966 - 2010 September) and PsycINFO (1806 - 2010 September). The search was conducted with the Medical Subject Headings (MeSH terms) ‘schizophrenia‘ or ‘psychotic disorder‘ or ‘psychosis‘, combined with ‘aggressive behavior‘ or ‘violence‘.

Inclusion criteria for studies considered were as follows:

• Samples including patients with schizophrena or schizoaffective disorder

• Patients in in-patient or in out-patient treatment

• any kind of aggressive behaviour towards others as an inclusion criterion

• informed consent required and mentioned

• patients actively involved in the respective study (i.e., not only consenting to chart review or gaining information from third persons)

• adult populations (≥ age 18 years)

• English language publications only

• publication in journals listed in the databases mentioned above

Exclusion criteria were populations in forensic settings, i.e., prisons, forensic psychiatry, or involuntary out-patient commitment. We excluded such studies following the assumption that incentives to participate in a study might be rather different for patients in those settings. Incentives like facilitations of imprisonment may have a stronger impact on the decision of forensic participants than on general psychiatric patients.

Outcomes

To assess external validity, we extracted the following data:

• inclusion criteria: diagnostic groups, gender, measures of violence

• information on non-participants

• information on drop-outs

• information on systematic comparisons of participants and non-participants

To gain further knowledge on the feasibility of recruitment according to our secondary objective, we extracted further data:

• proportion of drop-outs and patients reluctant to participate

• sample size

• period of recruitment, number of involved centres

• setting (in-patient, out-patient)

• requirements for study participants

• legal status of participants (voluntary/involuntary)

• incentives for participants

RESULTS

The literature search yielded 1419 hits (PubMed 1096, PsycINFO 323). The abstracts of all studies were screened with respect to the inclusion and exclusion criteria. In case of doubt, if no abstracts were available, the full texts were analysed by one of the authors (K.H.). In total, 33 studies fulfilled the defined inclusion criteria. There were several studies that based on the same data sets but dealt with different topics. In total, there were 20 independent data sets.

11 out of 33 studies investigated drug effects, 13 symptoms, prediction and course, 6 neurobiological issues, 2 subjective views and 1 a psychotherapeutic intervention. 19 of the studies included were published prior to the year 2000, 14 studies were published since then.

Study Inclusion Criteria

26 studies (78.8%) examined diagnostically homogenuous samples of patients with schizophrenia, while 7 studies (21.2%) had mixed samples comprised of patients with schizophrenia. 4 studies (12.1%) had samples with only male patients, and the others were comprised of both genders.

16 out of the 33 studies (48.5%) used recently observed or reported aggression as an inclusion criterion, while 10 used either recently observed aggression or aggression in the subjects' history, and 7 used a history of aggressive behaviour as an inclusion criterion. 5 of the studies (15.2%) defined a minimum score on a standardised scale as inclusion criterion, while most of the others used aggressive or violent incidents or aggressive behaviour without a clear definition.

Information on Non-Participants

In 6 out of 33 studies (18.2%), information on the percentage of patients who were identified by the screening procedure but refused to participate was provided. The percentage was 12.0% on average (sd = 10.7). Three out of 33 studies (9.1%) reported a systematic comparison of participants and non-participants, and only one reported detailed information on those patients, citing no significant difference [20].

16 out of 33 studies (48.5%) provided information on the percentage of drop-outs. Drop-outs were 22.6% on average (sd = 15.5). Four out of 33 studies reported characteristics of drop-outs.

Sample Size, Setting, Follow-Up, Legal Status, Incentives

The mean sample size was 93.5 (sd = 98.8). The subgroups of ‘aggressive‘ patients comprised the mean of 62.5 patients (sd = 46.0). 15 (45.5%) of the studies provided information on the recruitment period, which was 33.8 months on average (sd = 22.4). 28 out of 33 studies reported the number of participating centres, which was 2.0 on average (sd = 1.7). 29 studies (87.9%) recruited only in-patients, while 2 recruited in-patients and out-patients and 2 only out-patients. 10 out of 33 studies were cross-sectional in nature, and the others provided follow-up data of 16.3 weeks (sd 37.8) on average. Only one study [36] provided detailed information on the legal status of the participating patients. None of the 33 studies provided information on incentives given to patients for study participation.

A brief overview of the results is given in Table 1. For more details see Table 2 ^{(47KB, pdf)} .

Table 1.

Detected Clinical Studies on Aggression and Violence and Schizophrenia

Author(s)	Sample Size (Total/Aggr)	Standardized Definition Of Aggression	Information on Percentage of Non-Participants or Dropouts	Characteristics of Non-Participants or Dropouts	Comparison of Participants and Non-Participants
Cohen et al. [21]	126/126	X	- -	- -	- -
Tuason [22]	30/30	- -	Dropouts: 16,7%	X	- -
Tuason [23]	52/52	X	non-participants: 5,8% dropouts: 44,2%	- -	- -
Krakowski et al. [24]	44/44	- -	- -	- -	- -
Krakowski et al. [25]	89/55	- -	non-participants: Entire study cohort: - -; subgroup: 1,1% dropouts: Entire study cohort: - -; subgroup: 30,3%	- -	- -
Okuma et al. [26]	162/162	- -	Dropouts: 9,3%	- -	- -
Lapierre et al. [27]	61/31	- -	non-participants: 16,2% dropouts: 0,0%	- -	- -
Vartiainen et al. [28]	19/19	X	Dropouts: 31,6%	- -	- -
Allan et al. [29]	34/34	- -	Dropouts: 5,9%	- -	- -
Cheung et al. [30]	62/31	- -	- -	- -	- -
Cheung et al. [31]	62/31	- -	- -	- -	- -
Cheung et al. [32]	62/31	- -	- -	- -	- -
Cheung et al. [33]	62/31	- -	- -	- -	- -
Krakowski & Czobor [34]	137/75	- -	- -	- -	- -
Maguire et al. [35]	40/20	- -	- -	- -	- -
Sreenivasan et al. [36]	109/109	- -	- -	- -	- -
Hodgins et al. [37]	104/104	- -	Dropouts: Reassessed after 6 mon-89%, 12 mon-78%,18 mon-62%, 24 mon-46%	- -	- -
Krakowski et al. [38]	177/96	- -	- -	- -	- -
Nolan et al. [39]	51/26	- -	- -	- -	- -
Modai et al. [40]	56/26	- -	- -	- -	- -
Ritsner et al. [41]	55/26	- -	- -	- -	- -
Krakowski & Czobor [20]	246/246	- -	non-participants: 2,8% dropouts: 9,8%	X	X
Omerov et al. [42]	82/41	- -	- -	- -	- -
Arango et al. [43]	46/46	X	Dropouts: 10,9%	- -	- -
Barkan et al. [44]	40/20	- -	- -	- -	- -
Krakowski et al. [45]	110/110	- -	non-participants: 17,9% dropouts: 36,4%	- -	- -
Citrome et al. [46]	33/33	X	Dropouts: 39,4%	X	X
Kim et al. [47]	580/61	- -	Dropouts: Entire study cohort: - - Subgroup: 21,3%	- -	- -
Krakowski et al. [11]	110/110	- -	Dropouts: 9,1% no baseline, 13,0% no endpoint	X	X
Haddock et al. [48]	77/77	- -	non-participants: 28,7% dropouts: 6,5%; reassessed end of treatment: 92,2%, follow-up: 88,3%	- -	- -
Kim et al. [49]	103/46	- -	- -	- -	- -
Krakowski et al. [50]	93/93	- -	Dropouts: 26,9%	- -	- -
Nolan et al. [51]	44/44	- -	- -	- -	- -

Comparison of Quality of Reporting: Pharmacological Studies and Recent Publications

A descriptive comparison of pharmacological versus non-pharmacological studies shows that in 5 (45.5%) of the 11 pharmacological studies and none (0.0%) of the 23 non-pharmacologic studies violence or aggression were defined by a cut-off score on an aggression scale. Rates of non-participation were reported in 2 (18.2%) of the pharmacological and in 4 (17.4%) of the non-pharmacological studies. The number of drop-outs was reported in 9 (81.8%) of the pharmacological and in 7 (30.4%) of the non-pharmacological studies. A systematic comparison of participants and non-participants was conducted in 2 (18.2%) of the pharmacological versus in 1 (4.3%) of the non-pharmacological studies.

Comparing studies published in recent years (since 2000) to studies published before 2000, it turns out that in 2 (14.3%) of 14 recent publications and 3 (15.8%) of 19 older publications a definition of violence by a cut-off score was given. Rates of non-participation were reported in 3 (21.4%) of the recent publications and 3 (15.8%) of the older publications. The number of drop-outs was reported in 8 (57.1%) of the recent publications and 8 (42.1%) of the older publications. A systematic comparison of participants and non-participants was conducted in 3 (21.4%) of the recent publications and none (0.0%) of the older publications.

DISCUSSION

The primary objective of this systematic literature review was to investigate the external validity of published clinical studies on aggressive and violent behaviour in people with schizophrenia. Thus we focused our inclusion criteria on studies which required informed consent, required patient’s active participation and required aggressive behavior as an inclusion criterion. Though there is plenty of literature on violence in people with schizophrenia, only a limited number of 33 studies, based on 20 independent data sets, fulfilled the criteria.

To assess the external validity, we analysed whether criteria for study inclusion were clearly defined and whether appropriate information was given on drop-outs during the respective study, as well as on patients who had been screened for participation but were reluctant to participate. We found that only 15% of the studies that fulfilled our inclusion criteria for the review used inclusion criteria defined a priori by a minimum score on an aggression scale such as the Modified Overt Aggression Scale [52]. This finding is in line with a recent overview on mood stabilisers in aggressive patients with schizophrenia where the authors criticised a lack of transparency regarding the inclusion criteria and definitions of aggression [16]. The difficulty in recruiting patients for such studies is shown by the study description of Citrome et al. [46], where the inclusion threshold had to be lowered during the study because after 15 months only 8 patients had been recruited, then also allowing for the inclusion of milder forms of aggression. However, the inclusion of very mild or not clearly defined forms of aggressive behaviour makes the external validity of the respective studies questionable. This does not only concern the severity, but also the causes of aggression. Only 8 out of 33 studies included made some differentiation between reactive, psychopathology-related and other forms of violence, e.g., violence related to antisocial personality traits or intoxication.

An analysis and comparison of non-participants and participants was reported in only one study, and characteristics of drop-outs were reported only in 4 studies, though their percentage (where it was reported) was about 25% of the study population. Even if detailed information on non-participants may not be accessible due to ethical limits, it would be helpful to give some basic information on these patients. For futher investigations it is essential to know which subgroups of patients are likely to participate or not. Altogether, the external validity of most studies on aggressive behaviour in schizophrenia seems questionable, and sample selection bias is rather probable in the sense that patients with severe violent behaviour and very little cooperation – the most difficult to treat and manage – were not included. This is well in line with previous studies on non-participants in studies on patients with mental illness. Non-participants tend to be male, unmarried, and to have a more severe course of illness. They also typically tend to have a low socioeconomic status, co-morbid substance abuse problems and a history of violence [53-55]. Patients with schizophrenia are more frequently reluctant to participate in studies than patients with bipolar disorder [55, 56]. Among patients with schizophrenia, non-participation is associated with the disorganised subtype [54], severe positive symptoms and more frequent hospitalisation [56]. Additionaly, we compared the reporting quality in pharmacological and non-phamacological studies. Although pharmacological studies overall showed a better reporting quality that may be due to stricter study protocols, there is even a need to improve the reporting quality. Comparing studies published prior to the year 2000 and more recent publications did not yield a remarkable increase in the quality of reporting.

Our second objective was to elucidate difficulties in recruiting appropriate study samples among aggressive and violent patients with schizophrenia, which is important for the planning of future studies. Unfortunately, most of the studies did not report many of the relevant data, such as percentages of non-participants. Incomplete reporting of such essential study data is common in clinical studies and randomised controlled trials (RCT) with regard to patients with schizophrenia [57, 58]. Where reported, only about 12% of eligible patients had refused to participate. This seems to be a small proportion; however, in many studies, the requirement for patients was only participation in an interview, participation in an RCT on medication, or consent to taking blood specimens for genetic tests. There is a remarkable discrepancy between the huge body of literature on predictors and associated factors of violence in schizophrenia and the nearly complete lack of evaluated intervention programmes, with only one exception [48], which might indicate the difficulty of conducting such studies. Not only might it be difficult to recruit patients, but bias emerges as the result of a considerable percentage of drop-outs during the study, which was 22% in the studies detected here (where information was reported at all). This was still less than the 40% in RCTs Gilbody et al. [58] reported in an overview of schizophrenia studies. If all types of studies on schizophrenia were included, this percentage would be much smaller (12%) [59].

The mean sample size was 93, more than the reported average of studies in schizophrenia in general [59], but the well-known tendency of studies to be underpowered is found here, too. The vast majority of studies used in-patients as samples. This can be understood as an indirect hint at the difficulties of conducting clinical studies outside the hospital setting with patients who have displayed violent behaviour. However, this can also be a further source of bias, because the factors associated with in-patient violence are rather different from those associated with community violence [3]. The recruitment period, if reported at all, was about 3 years on average, which may be a further hint at the difficulities in realising such studies. If recruitment of patients for studies is difficult and, in addition, it is difficult to keep patients to study protocols, financial incentives for participants are helpful and frequently used [60] but rarely reported [61]. Unfortunately, none of the detected studies reported on incentives for participants. A better reporting quality would not only give helpful hints for recruitment in future studies, but is part of scientific ethics. Motivated by one of the reviewers of this paper, we tried to contact the principal investigators of the reviewed studies via e-mail, as indexed in the SCOPUS database, concerning the question of incentives for participating patients. However, a considerable proportion of the studies is more than ten years old and some of the authors had moved to other institutions or retired, from others we did not receive an answer. The resulting picture is incomplete but suggests that incentives of different kind may have been given in most cases. Financial compensation was described as small amounts of about 10 $ or €. Other incentives could be transferral to a special research unit with better staffing and more convenient conditions of hospitalization.

Given the difficulties in recruiting aggressive patients mentioned above, it is questionable that studies based on informed consent of patients are representative for the nature and severity of aggression in schizophrenia. It has to be assumed that the most aggressive and ill patients are mostly reluctant to participate in studies. Therefore, findings from clinical studies, particularly RCTs, have to be supplemented by data from other sources, e.g. observational and epidemiological studies, to allow generalisability to clinical practice and to improve external validity.

A limitation of this literature review is that we confined our search to MeSH terms and to studies published in English and available in PubMed or PsycINFO. However, there is little reason to assume that other studies not fulfilling these criteria are better reported in terms of study designs and outcomes.

CONCLUSION

Most studies on violence in patients with schizophrenia are poorly reported in terms of external validity, and are questionable with regard to generalisability. For future studies, it is essential to report the following methods and outcomes: inclusion criteria, especially a clear definition of aggression or violence on an appropriate scale; number of patients screened for inclusion; number and characteristics of participating and non-participating patients, and comparison of these groups; number and characteristics of finishers and drop-outs, and a comparison of these groups, especially in terms of violent behaviour at inclusion; and, intention-to-treat analyses. In addition, information on consent procedures and incentives should be reported. Table 3 summarizes the recommendations for improving reporting quality.

Table 3.

Recommendations for Improving Reporting Quality

Recommendations for Reporting in Future Studies

Inclusion criteria: clear definition of aggression by cut-off score of an appropriate scale Number of patients screened for inclusion Number and characteristics of participating and non-participating patients Comparison of participating an non-participating patients Number and characteristics of finishers and drop-outs Comparison of finishers and drop-outs (esp. in terms of violent behaviour) Intent-to-treat-analysis Information on consent procedures and incentives

SUPPLEMENTARY MATERIAL

Table 2 ^{(47KB, pdf)} Detailed overview of detected clinical studies on aggression and violence and schizophrenia.

CONFLICT OF INTEREST

The authors confirm that this article content has no conflicts of interest.