The Relationship Between Level of Training and Accuracy of Violence Risk Assessment

Abstract

Objective

Although clinical training programs aspire to develop competency in violence risk assessment, little research has examined whether level of training is associated with the accuracy of clinicians’ evaluations of violence potential. This is the first study to compare the accuracy of risk assessments by experienced psychiatrists to those of psychiatric residents. It also examined the potential of a structured decision support tool to improve residents’ violence risk assessments.

Methods

Using a retrospective case control design, medical records were reviewed for 151 patients who assaulted staff at a county hospital and 150 comparison patients. At admission, violence risk assessments had been completed by psychiatric residents (N= 38) for 52 patients, and by attending psychiatrists (N = 41) for 249 patients. Trained, blinded research clinicians coded information available at hospital admission with a structured risk assessment tool, the HCR-20 Clinical (HCR-20-C) scale.

Results

Receiver operating characteristic analyses showed that clinical estimates of violence risk by attending psychiatrists had significantly higher predictive validity than those of psychiatric residents. Risk assessments by attending psychiatrists were moderately accurate (AUC = .70), whereas risk assessments by residents were no better than chance (AUC = .52). Incremental validity analyses showed that addition of information from the HCR-20-C had the potential to improve the accuracy of risk assessments by residents to a level (AUC = .67) close to that of attending psychiatrists.

Conclusions

Less training and experience is associated with inaccurate violence risk assessment. Structured methods hold promise for improving training in risk assessment for violence.

Although most people with mental illness are not violent, they are at greater risk of violence than the general population (1–3). In settings such as psychiatric inpatient units and emergency rooms, violence is a common problem that is a frequent cause of injuries to clinicians (4–6). Trainees in psychiatry, clinical psychology, and other mental health disciplines often complete rotations in these acute settings and are especially vulnerable to being victims of patient aggression (7–10).

Credentialing organizations and the public expect psychiatrists, clinical psychologists, and other mental health professionals to have competency in violence risk assessment (11,12). Yet surveys show that clinical trainees often report that their education is insufficient in the area of violence risk assessment and management (13–15). Moreover, there is surprisingly little scientific evidence that training and experience confer an advantage in the ability to assess patients’ risk of violence. In fact, one widely-cited study challenged this assumption. Quinsey and Ambtman (16) provided case files to a group of experienced forensic psychiatrists and a group of high school teachers and asked them to evaluate the likelihood that the patients would become violent; the results suggested that the reliability and validity of judgments by the two groups did not differ. Although raising important issues, the generalizability of conclusions based on that study may have been limited by use of case files rather than live clinical encounters. Research is lacking on whether, in the course of service delivery to patients, highly trained and experienced clinicians are more accurate than those with little training in their violence risk assessments. The primary aim of this study was to evaluate that issue.

If trainees are indeed less able to accurately perform risk assessments for violence, it is important to consider ways to improve their accuracy. One promising strategy in this area is educating them in structured methods. In recent years, a variety of structured risk assessment measures have been developed that have the potential to improve violence risk assessments by prompting clinicians to consider patient characteristics supported by research as risk factors for violence (17,18). That is, they are evidence-based decision aids. One such tool, the Historical, Clinical, Risk Management-20 (HCR-20), asks the evaluator to consider historical, clinical, and situational variables linked to violence potential (19). Many studies have supported the validity of the HCR-20 as an aid to violence risk assessment (20–22). Despite the accumulating research supporting the predictive validity of structured methods of violence risk assessment, these validated measures are relatively new and are only beginning to be adopted in many areas of clinical practice (21,23). A secondary goal of the present study was to explore the potential of a structured risk assessment instrument to improve the predictive validity of trainees’ risk assessments.

In sum, the present study addressed the following questions based on review of medical records of a sample of hospitalized patients. First, were the violence risk assessments that had been completed at the time of admission by attending psychiatrists more accurate than those that had been completed by psychiatric residents? Second, would addition of information from the HCR-20 improve the accuracy of risk assessments above that of residents’ unstructured clinical evaluations of violence potential?

Methods

The study employed a retrospective, case control design. The protocol was approved by the Committee on Human Research of the University of California, San Francisco.

Participants and Setting

The setting was the four locked psychiatric inpatient units of the county hospital that serves a city of approximately 750,000 residents. We reviewed incident reports of all unduplicated patients (N = 172) who had physically assaulted staff between January 1, 2003 and December 31, 2008. For patients who had more than more incident of violence, only the first incident was considered. To verify the severity of the episodes of aggressive behavior, descriptions of them in the incident reports and medical records were rated by two doctoral level mental health clinicians using a standardized measure of inpatient aggression, the Staff Observation Aggression Scale – Revised (SOAS-R) (24). The interrater reliability of these ratings was high (kappa = .93). The mean± SD SOAS-R total score was 17.3± 2.3, well above the threshold for severe aggression of 9.0 recommended an author of that scale (25). After matching for psychiatric inpatient unit and month of admission, 173 nonviolent comparison patients were selected using a table of random numbers generated by Excel. Data were excluded for 21 of the violent patients and 23 of the nonviolent patients because their medical records did not include information about level of training of the physician who completed the admission risk assessment, resulting in a final sample of 151 violent patients and 150 nonviolent patients.

Clinical Risk Assessment Ratings

At admission, physicians had rated each patient on a four-point assault precaution checklist that ranged from zero (no clinical indication for precautions) to three (strong intent is present or unable to control impulses). For each patient, only one clinician made an assessment of violence risk.

Thirty-eight psychiatry residents rated 52 patients. The residents had a mean±SD of 1.2± 1.0 years of residency training and a mean age of 30.7±2.5 years. The mean number of patients rated by each resident was 1.4± .7, with a range of 1 – 4 patients. Forty-one attending psychiatrists rated 249 patients. The attendings had a mean of 13.7±7.1 years of post-residency experience (based on data available for 199/249 patients) and a mean age of 46.8±7.5 years. The mean number of patients rated by each attending was 6.1± 8.3, with a range of 1 – 34 patients.

Structured Risk Assessment Ratings

The HCR-20 Clinical subscale (HCR-20-C) (19) was used to rate charts for acute risk of violence. The Clinical subscale includes five items (active symptoms of major mental illness, impulsivity, lack of insight, negative attitudes, and unresponsive to treatment) that are rated on a three-point scale: 0 (absent), 1 (possibly present), and 2 (present). The Clinical scale was selected for use based on previous research evidence that it has a stronger association with short term risk of aggression by acute psychiatric inpatients than the other components of the HCR-20 (26,27). Several previous studies have rated HCR-20 items based on record review (26–28), although interviews may also be used to supplement file information.

In this study, two psychiatric nurses, who had been trained in the use of the measure, rated the HCR-20-C based on information available in the medical records at the time of admission. The raters were blind regarding whether or not the patients later became violent. Their inter-rater agreement, as measured by the intraclass correlation coefficient (ICC), was .78 on eight practice charts and .81 on a sample of 14% (43/301) of study participants.

Data Analysis

To characterize the sample, comparisons of whether patients evaluated by residents and attendings had different demographic and clinical characteristics were conducted using chi-square analyses for categorical variables and t tests for continuous variables.

The Area under the Curve (AUC) of the Receiver Operating Characteristic (ROC) was used to assess the predictive validity of the risk assessments (29,30). ROC analysis describes predictive accuracy across a range of cutoff scores and is less dependent on the base rate of violence than conventional approaches. The AUC represents the probability that a randomly selected violent patient will have been evaluated by the risk assessment method as at higher risk than a randomly selected nonviolent patient. The AUC can range from 0 (perfect negative prediction) to 1 (perfect positive prediction), with .5 representing accuracy no better than chance. As a general rule, the predictive validity of AUCs of .80 to .90 is considered excellent, .70 to .79 acceptable, and .60 to .69 modest (31). The AUC for patients evaluated by residents was compared with the AUC for patients evaluated by attending psychiatrists with a Wald test for the difference in AUC for two independent groups (29).

To address the secondary research question regarding the potential of the HCR-20-C to improve predictive accuracy over and above residents’ unstructured clinical risk assessments, we performed incremental validity analyses following a method previously described (32). Specifically, we performed two linear regression analyses whereby the clinical risk assessment ratings were used to predict the HCR-20-C scores, and the HCR-20-C scores were used to predict the clinical risk assessment ratings. The studentized residuals from these analyses represent the unique independent variance of each measure beyond that of the other (i.e., the variance of the clinical risk assessment independent of the HCR-20-C and vice versa). We then conducted ROC analyses based on these residuals.

Data were analyzed using SAS Version 9.2, Stata Version 10, and PASW Version 18.

Results

Patient Characteristics

Overall, the study group tended to be an ethnically diverse, middle-aged group of men with a psychotic disorder (Table 1). The patients assessed by residents and attending psychiatrists showed no significant differences in demographic and clinical characteristics. Similarly, the proportion of patients who became violent in the hospital did not differ significantly between patients who had been evaluated at admission by residents or attendings.

TABLE 1.

Characteristics of Patients Evaluated for Violence Risk at Admission by Psychiatric Residents (N=52) and Attending Psychiatrists (N=249)^a

Characteristic	Resident		Attending		Analyses
	M±SD		M±SD		t	df	p
Age	42.42±13.64		43.22±14.69		.36	299	.72
GAF at Admission	27.93±6.43		26.65±7.29		1.07	271	.29
	N	%	N	%	χ2	df	p
Gender					.05	1	.83
Male	33	63	154	62
Female	19	37	95	38
Ethnic Background					5.69	5	.34
Caucasian	29	56	45	18
African American	11	21	66	27
Asian	6	12	30	12
Hispanic	6	12	100	40
Other	0	0	8	3
Legal Status					1.01	2	.61
Voluntary	0	0	4	2
Involuntary civil commitment	51	98	242	97
Other/Unknown	1	2	3	1
Axis I Diagnosis at Admissionb
Psychotic disorder	44	85	210	84	.03	1	.96
Mood disorder	6	12	32	13	.07	1	.80
Substance use disorder	13	25	50	20	.63	1	.43
History of Violence in Two Weeks
Prior to Admission	12	23	36	14	2.22	1	.14
Study Group					.51	1	.82
Violent during hospitalization	27	52	124	50
Not violent during hospitalization (control group)	25	48	125	50

^aPercentages are for the proportion of patients without missing data. For evaluations by residents and attendings, respectively, data were available for 42/52 (81%) and 231/249 (93%) of patients on the Global Assessment of Functioning (GAF) scale; and data were available for 51/52 (98%) and 240/249 (96%) of patients, respectively, regarding history of preadmission violence.

^bMultiple diagnoses possible for each patient due to comorbidity.

Relationship Between Level of Training and Accuracy

Figure 1 shows that clinical risk assessments by attending psychiatrists had a moderate degree of predictive validity (AUC = .70), whereas those completed by residents were no better than chance (AUC = .52). The risk assessments by attending psychiatrists were significantly more accurate than risk assessments by residents (χ² = 5.52, df = 1, p = .02).

Figure 1.

Receiver Operating Characteristic (ROC) Analysis Showing Predictive Validity of Clinical Estimates of Violence Risk of Patients Assessed by Psychiatric Residents (N=52) and Attending Psychiatrists (N=249)a

^aArea Under Curve (AUC): Attending psychiatrist (AUC = .70); Psychiatric resident (AUC = .52)

Incremental Validity of the HCR-20-C

For the patients evaluated by residents, the AUC of the HCR-20-C was .60 (SE = .08). ROC analysis of the studentized residuals showed that the AUC for the variance attributable to the clinical risk estimates independent of the HCR-20-C was only .55 (SE = .08), whereas the variance attributable to the HCR-20-C independent of the clinical risk estimates was more strongly associated with violence (AUC = .67, SE = .08). Figure 2 illustrates the application of the HCR-20-C to the patients evaluated by psychiatric residents.

Figure 2.

Receiver Operating Characteristic (ROC) Analysis Showing Predicative Validity of the HCR-20-C, After Controlling for Residents’ Clinical Estimates of Violence Risk (N=52)a

^aArea Under Curve (AUC): HCR-20-C after controlling for residents’ clinical risk assessment (AUC = .67)

It is of interest to note that the AUC of the HCR-20-C independent of the residents’ clinical risk estimates (.67) was similar to the AUC of the clinical risk estimates of the attending psychiatrists (.70). This suggests that adding information from the HCR-20-C to residents’ evaluations has the potential to bring the accuracy of their risk assessments close to that of more experienced attending psychiatrists.

For the patients evaluated by attending psychiatrists, the AUC of the HCR-20-C was .69 (SE = .03). Incremental validity analysis showed that the AUC for the variance attributable to the attending psychiatrists’ clinical risk estimates independent of the HCR-20-C was .65 (SE = .04), whereas the variance attributable to the HCR-20-C independent of the attendings’ clinical risk estimates was limited (AUC = .57; SE = .04). Hence, for patients evaluated by attending psychiatrists, the HCR-20-C yielded little incremental validity over that of the unstructured clinical risk estimates.

Discussion

To our knowledge, this is the first study to compare the predictive validity of assessments of violence risk by experienced clinicians to those made by trainees in a clinical setting. The results support the conclusion that level of training is associated with greater accuracy of violence risk assessment. Whereas the risk assessments of highly experienced attending psychiatrists demonstrated a moderate degree of predictive validity, those of a group of junior psychiatric residents—who evaluated patients with similar characteristics—were no better than chance.

This study has implications for training and education regarding violence risk assessment. First, it highlights the importance of education in violence risk assessment to clinical training programs (33–37). Second, it provides further evidence that structured methods hold promise as an aid to development of skill in risk assessment. Previous research has shown that when trained in the HCR-20, psychiatric residents and learners from other backgrounds demonstrate improved clinical reasoning about violence risk as evidenced by improved documentation of case formulations and intervention plans (33. 38, 39). The present study shows that adding information from the HCR-20-C has the potential to improve trainees’ risk assessments to a level of predictive accuracy similar to that of experienced attending psychiatrists. Information needed to rate the HCR-20-C is routinely collected in the course of admitting patients to a psychiatric inpatient unit, and prompting trainees to rate this structured measure could aid them in attending to valid risk markers from among the vast amount of information that may become available during the course of an admission workup. Third, the improvements in accuracy offered by use of a structured risk assessment tool have the potential to guide development of interventions to prevent violence (40).

A strength of this study is its external validity. First, because of the large sample of attending psychiatrists and residents results are more likely to be generalizable. Second, the ecological validity of the study is enhanced by its evaluation of the outcome (violence) in a clinical setting undisturbed by the predictor variables (type of risk assessment method and level of training). Limitations of the study include that it was conducted in one site and relied on retrospective chart review. Use of incident reports to identify violent incidents may not have been sensitive to milder forms of patient aggression. It possible that the predictive validity of the HCR-20-C would have differed if the patients had been rated and followed prospectively. Also, the generalizability of the findings beyond the inpatient setting remains a topic for future research.

Conclusions

In summary, this study addresses an important gap in the field of violence risk assessment related to the relationship between level of training and accuracy of violence risk assessment. The results support the conclusion that level of training confers an advantage in the accuracy of risk assessments for violence. In addition, the results illustrate that structured methods have the potential to augment training in a way that may improve the accuracy of risk assessments.