Suicide risk assessment tools do not perform worse than clinical judgement

The study by Quinlivan and colleagues (1) could be interpreted to suggest that clinician and patient ratings are better than actuarial tools in predicting self-harm after an emergency hospital presentation with self-harm. However, we would argue that this is an incorrect interpretation.

First, the clinical evaluation appears to have occurred after these tools were completed by the same clinician, and, although they were masked to the overall score, the clinical impression will therefore have been strongly informed by the items in these suicide risk assessment tools. In fact, the study does not appear to be a comparison between actuarial tools and a distinct, unstructured clinical judgement, but a comparison between actuarial tools and what is called a structured clinical judgement approach (where structured questions about relevant risk factors are asked, and then a clinical judgement is made about an individual’s overall risk level). Clarification of the exact procedure used is important for interpreting the findings.

Second, the authors correctly point out that, on the basis of ROC curves, risk assessment tools performed no better than clinician ratings. The other way of looking at this, however, is that clinician ratings performed no better than risk scales. In particular, the Manchester Self-Harm Rule (2), a 4-item tool, performed just as well. Importantly, the authors found no evidence of between-hospital heterogeneity for this tool’s performance. Clinician ratings on the other hand showed substantial heterogeneity between hospitals, with specificity ranging from 58% to 82%. The lack of variability in the actuarial tools could be argued to be an advantage when performance between clinician rating and assessment tool is no different. Furthermore, tools like this will be considerably quicker, leaving more clinician time for risk management (as opposed to assessment).

Third, the clinicians were based in teaching hospitals (Brighton, Bristol, Derby, Manchester, and Oxford) with longstanding research interests in self-harm. Whether the reported predictive accuracy of clinician ratings is generalizable to non-specialist centres is an empirical question.

Fourth, the patient rating may also have been influenced by the questions asked by the tools (which tend to be categorical and therefore easy to work out what constitutes a risk factor). In a sense, then, the patient rating is a form of structured judgement.

Comparing risk tools with clinicians may not be informative, or even feasible, as clinical interviews already include many of the items used in risk tools. Instead, future research should compare actuarial scores with or without additional clinician input. In other words, if clinicians disagree with the risk level provided by actuarial tools, does this reclassification lead to an improvement in predictive performance? As the AUCs for the tools in this study ranged from 0.55 to 0.72, there may be considerable room for improvement by incorporating novel and modifiable risk factors as has been shown in violence risk assessment in patients with severe mental illness (3). Ultimately, randomized studies will be required to establish the effects of different approaches to risk assessment on patient and service outcomes.