Pokorny's complaint: the insoluble problem of the overwhelming number of false positives generated by suicide risk assessment

Abstract

Alex Pokorny's 1983 prospective study of suicide found that 96.3% of high-risk predictions were false positives, and that more than half of the suicides occurred in the low-risk group and were hence false negatives. All subsequent prospective studies, including the recent US Army Study To Assess Risk and Resilience in Servicemembers (STARRS), have reported similar results. We argue that since risk assessment cannot be a practical basis for interventions aimed at reducing suicide, the alternative is for mental health services to carefully consider what amounts to an adequate standard of care, and to adopt the universal precaution of attempting to provide that to all of our patients.

Pokorny's pioneering study

It is over 30 years since the publication of Alex Pokorny's landmark prospective study of suicide among consecutive first admissions to the Houston Veterans Administration Medical Center.^1,2 Pokorny examined a cohort of 4800 men using a comprehensive array of relevant and reliable rating scales and assessed 803 (17%) to be at high risk of suicide because of the presence of a combination of risk factors. In the next 5 years, 30 (3.7%) high-risk patients and 37 (0.9%) of the 3997 lower-risk patients died by suicide, an odds of suicide that was 4 times greater in the high-risk group.

The 37 suicides among lower-risk patients were in effect false negatives. Pokorny considered false negatives to be inevitable because patients sometimes conceal their suicidal plans and their circumstances change over time. What concerned him more was the overwhelming proportion of false positives, because 96.3% of the patients categorised as high-risk did not commit suicide. He concluded:

‘We might tolerate 50% false negatives; if we could apply a screening test that would correctly identify only half of the future suicides without false positives that would be very helpful. However, with currently known tests to identify the actual suicides, we will also have to make a great many false-positive identifications, labeling up to a quarter of the total group as future suicides when only 1% to 5% actually are. From a cost-benefit standpoint, the application of such a test is simply not feasible.’¹

Other prospective studies of suicide

There have been relatively few prospective studies of the factors associated with subsequent suicide, probably because of the methodological and logistical difficulties involved. Not long after Pokorny, Goldstein et al³ followed up a cohort of 1906 patients diagnosed with affective disorders and found that none of the 46 suicides occurred among the group, albeit smaller than in Pokorny's study, defined as being at high risk. More recently, two prospective studies examined the proportion of suicide deaths among those considered to be at high risk.^4–6 Madsen et al^4,5 reported a suicide rate of 0.14% among those defined as high risk in a national study of the suicide of psychiatric in-patients in Denmark, and Steeg reported the suicide of 0.5% of patients identified as being at high risk among a large cohort of people who presented to emergency departments in three English counties after self-harm.⁶

Faced with growing concern about suicides among veterans and current serving members of the U.S. Armed Forces, the U.S. Department of Veterans Affairs and the U.S. Department of Defense included the use of risk stratification in clinical practice guidelines.⁷ As part of the Study to Assess Risk and Resilience in Servicemembers (STARRS), Kessler and associates then examined factors associated with suicide among 53 769 American soldiers in the 12 months after discharge from military psychiatric hospitals,⁸ with the benefit of the very detailed longitudinal US Army personnel database and highly sophisticated statistical techniques derived from artificial intelligence research. They identified a high-risk group comprising 2689 admissions, or 5% of the sample, from which there were 36 suicides, corresponding to a 22 times increased risk of suicide in that group. However, in other respects the results were eerily familiar, as nearly half of the suicides occurred among the 51 080 low-risk patients, and only 1.3% of the high-risk group died by suicide, leaving an overwhelming number of false positive cases.

The problem remains of the disturbingly high suicide rates among psychiatric patients as a whole. For example, in the Madsen study, the rate of suicide of psychiatric in-patients was 72 times that of the general community, at 860 per 100 000 patient years, and in the Steeg study the suicide rate among all patients presenting to hospital after self-harm was 622 per 100 000 patient years. Hence, although we know that all of the patients under our care are at greatly increased risk of suicide compared with the wider community,⁵ our ability to distinguish between groups of patients with respect to the probability of subsequent suicide risk is, at best, quite modest.

Implications for mental health services

This observation has important implications for mental health services, given the unrealistic expectations for risk assessment to prevent serious adverse events, and the way risk assessment has changed professional practice.⁹ If risk stratification is to be a part of how mental health services approach suicide prevention, we need to carefully consider the interventions offered to patients believed to be at high risk but denied to those assessed to be at lower risk. Because the vast majority of high-risk patients will not die by suicide, any intervention has to be shown to be somewhat effective, but also benign enough so as not to infringe on the rights of the many with false positive assessments. However, if we do have an effective and benign intervention, it is then hard to justify denying this to low-risk patients, who, after all, are still many times more likely to die by suicide than the general community.

If there is no effective and yet benign intervention to justify offering different treatments to groups of patients on the basis of risk stratification, how can mental health services respond to the knowledge that our patients are at greatly increased risk of suicide compared with the wider community?

The alternative is the application of universal precautions to all patients, including the design of in-patient facilities that minimise the opportunity to attempt suicide,¹⁰ the provision of community treatment for all patients in the weeks after discharge from hospital,¹¹ and safety planning at the point of contact in emergency departments.¹² Most importantly, every patient should have access to timely, individualised, high-quality treatment for psychiatric disorder. Modifiable factors associated with suicide should be addressed in all patients, not only those considered to be at greatest risk. Moreover, no intervention can be justified on the basis of the assessed risk alone. We go so far as to suggest that the assessed risk of suicide on its own is not a sufficient reason for restrictive interventions such as involuntary detention in hospital or other coercive treatment,^13,14 which would then need to be justified on additional grounds, including the patient's inability to recognise the potential benefit of treatment and their particular circumstances and treatment needs. In any case, preventive detention in hospital of all patients assessed to be at high risk of suicide would be impossible, because of the huge numbers of patients identified and the long duration of secure detention that would be required to protect such patients.

Addressing the modifiable risk factors in populations of patients involves ensuring the adequate identification, assessment and treatment of patients with a range of mental disorders, addressing substance use and, wherever possible, limiting the availability of potentially lethal means to die by suicide. Treatment should be with the patient's consent, or provided on the basis that the patient lacks the capacity to consent, rather than being based on perceived risk, which we now know we are not able to reliably assess.

A further implication for mental health services of the demonstrated limitation of risk assessment is in responding to lawsuits for failing to predict the suicides of individual patients. While the suicide of any patient might be foreseeable in the legal sense of being not fanciful or far-fetched, mental health services cannot be reasonably expected to be able to identify which patients will die by suicide. Expert evidence in one recent case on the estimated probability of suicide after discharge included absurdly high estimates of between a 5 and 70% chance of suicide within days of discharge.¹⁵ Instead of guessing the probability with the help of hindsight bias,¹⁶ negligence claims after suicides should hinge on the accepted standard for care of people with various psychiatric disorders, regardless of the presence of known risk factors. Services should be concerned about the adverse consequences of failing to provide an adequate standard of care to any patient they assess or treat, not only those considered to be at high risk.

Pokorny's complaint that the overwhelming number of false positives renders suicide risk assessment unfeasible is just as valid in 2016 as it was in 1983. His finding, which has been replicated in all subsequent studies, poses a challenge to military and civilian mental health services that have been developed around a model of identifying and managing risk. In response, we recommend abandoning attempts to design interventions based on risk stratification and instead aim to provide an adequate standard of care to all of our patients.