How Should We Reconcile Statistical and Clinician Predictions of Suicide Risk

Abstract

Statistical models, often based on electronic health records (EHRs), can accurately identify patients at high risk of suicide attempt or death, leading to health system implementation of risk prediction models as a component of population-based suicide prevention. At the same time, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from those two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting regarding Suicide Risk Algorithm Applications in Healthcare Setting. Participants in that meeting here summarize key issues regarding the potential clinical application of prediction models. This report attempts to clarify the key conceptual and technical differences between traditional clinician risk prediction and predictions from statistical models, reviews the limited evidence available regarding both the accuracy of and concordance between these alternative methods of prediction, presents a conceptual framework for understanding agreement and disagreement between clinician and statistical and clinician prediction, and identifies priorities for improving data regarding suicide risk as well as priority questions for future research. The future of suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of those methods.

Introduction

Despite increasing attention to suicide prevention, US suicide rates continue to rise – increasing by over one-third since 1999¹. Over three-quarters of people attempting suicide or dying by suicide had some outpatient healthcare visit during the prior year ^{2, 3}. Healthcare encounters are a natural setting for identifying and addressing risk of suicidal behavior⁴, but effective prevention requires accurate identification of risk. Unfortunately, assessment based on traditional risk factors is not accurate enough to direct interventions to people at highest risk^{5, 6}. Recent research demonstrates that statistical models, often based on electronic health records (EHRs), more accurately identify patients at high risk in mental health, general medical, and emergency department settings^7–12. Motivated by that research, some healthcare systems have implemented risk prediction models as a component of population-based suicide prevention¹³. At the same time, some have questioned whether statistical predictions can really inform clinician decisions given concerns regarding high false-positive rates, potential for false labelling, and lack of transparency or interpretability¹⁴.

Current ¹³ and planned implementations of suicide risk prediction models typically call for delivering statistical predictions to treating clinicians. Clinicians then assess risk and implement specific prevention strategies (e.g., collaborative safety planning¹⁵) or treatments (e.g., Cognitive-Behavioral Therapy¹⁶ or Dialectical Behavior Therapy¹⁷). This sequence presumes that clinician assessment improves on a statistical prediction. Whether or not a clinician assessment adds to statistical risk predictions, however, depends on whether those two methods are competing, complementary, or simply duplicative.

In June 2019, the National Institute of Mental Health convened a meeting regarding Suicide Risk Algorithm Applications in Healthcare Settings²³. That meeting considered a range of clinical, technical, and ethical issues regarding implementation of prediction models. As participants in that meeting, we here present our conclusions and recommendations regarding one central issue: how algorithms or statistical predictions might complement, replace, or compete with traditional clinician prediction. First, we attempt to clarify conceptual and practical differences between clinician prediction and predictions from statistical models. Second, we review the limited evidence available regarding accuracy of and concordance between these alternative methods. Third, we present a conceptual framework for understanding agreement and disagreement between clinician and statistical predictions. Finally, we identify priorities for improving data regarding suicide risk and priority questions for research.

While we focus prediction of suicidal behavior, clinicians can expect to encounter more frequent use of statistical models to predict other clinical events, such as psychiatric hospitalization²⁴ or response to antidepressant treatment²⁵. We hope the framework we describe can inform research and policy regarding the integration of statistical prediction and clinician judgment to improve mental health care.

Defining and Distinguishing Statistical and Clinician Prediction

We define statistical prediction as the use of empirically derived statistical models to predict or stratify risk of suicidal behavior. Those models might be developed using traditional regression or various statistical learning methods, but the defining characteristic is the development of a statistical model from a large dataset with the aim of applying that model to future practice. We define clinician prediction as a treating clinician’s estimate of risk during a clinical encounter. As summarized in Table 1, statistical and clinician predictions typically differ in the predictors considered, the process for developing predictions, the nature of the resulting prediction, and the practicalities of implementation.

Table 1 –

Contrasting statistical and clinical prediction of suicide risk

DISTINGUISHING CHARACTERISTICS	STATISTICAL PREDICTION	CLINICAL PREDICTION
INPUTS
Data sources	EHRs and other computerized databases	Clinical observations, clinician review of available records
Number of possible predictors	Hundreds or thousands	Typically fewer than 10
PROCESS
Combining and weighting predictors	Statistical optimization, often involving machine learning	Clinician judgment regarding relevance and weighting of risk factors
Accommodating heterogeneity	Interactions or tree-based approaches to identify subgroup-specific predictors	Clinician judgment regarding applicability of specific risk factors to individuals
Balancing sensitivity and specificity	Explicit assessment of performance at varying thresholds (May also evaluate alternative loss or cost functions)	Clinician judgment regarding importance of false positive and false negative errors
OUTPUTS
Product	Unidimensional prediction – often a continuous score	Clinical formulation
Use in treatment decisions	Not intended to identify causal relationships or treatment targets	Aims to identify causal relationships and treatment targets
IMPLEMENTATION
Timing	Typically computed prior to clinical encounters	Formulated during clinical encounters
Scale	Batch calculation for large populations	Distinct assessments for each individual

Inputs for generating predictions

Data available to statistical models are both broader and narrower than data available to human clinicians. Statistical models to predict suicidal behavior typically consider hundreds or thousands of potential predictors extracted from health records data (EHRs, insurance claims, hospital discharge data) ²⁶. Those data most often include diagnoses assigned, prescriptions filled, and types of services used, but could also include information extracted from clinical text or laboratory data^{27, 28}. They may also consider “big data” not typically included in health records, such as genomic information, environmental data, online search data, or social media postings^{29, 30}. In contrast, clinician reasoning is typically limited to a handful of predictors rather than hundreds or thousands. Clinicians, however, can consider any data available during a clinical encounter, ranging from unstructured clinical interviews to standardized assessments tools^{21, 31–33}. Most important, clinician assessment usually considers narrative data systematically recorded in EHRs, such as stressful life events, and subjective information such as clinicians’ observations of thought process or nonverbal cues^{34, 35}.

Process for generating predictions

Clinician and statistical prediction differ fundamentally in accommodating high dimensionality and heterogeneity of predictive relationships. Statistical methods select among large numbers of correlated predictors and assign appropriate weights in ways that a human clinician cannot easily replicate. Those statistical methods for dimensionality reduction do involve choices and assumptions that may be obscure to clinicians and patients. In contrast, clinician prediction relies on individual judgment to select salient risk factors for individuals and assign each factor greater or lesser importance. Statistical prediction reduces dimensionality empirically, while clinician prediction reduces dimensionality using theory, heuristics, and individual judgment.

Statistical models account for heterogeneity or complexity by allowing non-linear relationships and complex interactions between predictors. Clinician prediction may consider interactions among risk factors, but in a subjective rather than quantitative sense. For example, clinicians might be less reassured by absence of suicidal ideation in a patient with a history of unplanned suicide attempt.

Statistical predictions typically produce a continuous risk score rather than a dichotomous classification. Selecting a threshold on that continuum must explicitly balance false negative and false positive errors. A higher threshold (less sensitive, more specific) is appropriate when considering an intervention that is potentially coercive or harmful. Machine learning methods for developing risk scores also permit explicit choices regarding loss or cost functions that emphasize accuracy in different portions of the risk spectrum. While clinician prediction may consider the relative importance of false negative and false positive errors, that consideration is not usually explicit or quantitative.

Outputs or products

Statistical prediction typically has narrower goals than clinician prediction. Statistical models simply identifies associations in large samples and combines those associations to optimize a unidimensional prediction, probability of suicidal behavior during some specific time period. Resulting statistical models are optimized for prediction but often have little value for explanation or causal inference. Even when statistical models consider potentially modifiable (e.g. alcohol use) risk factors, statistical techniques used for prediction models are not well-suited to assess causality or mechanism. In contrast, clinician prediction considers psychological states or environmental factors to generate a multidimensional formulation of risk. Explanation and interpretation are central goals of clinician prediction.

Because statistical prediction values optimal prediction over interpretation, statistical models can paradoxically identify appropriate treatments (such as starting a mood stabilizer medication) as risk factors. Recent discussion underlines the importance of properly modeling interventions in prognostic models to prevent this very phenomenon³⁶. In contrast, clinician prediction often aims to identify modifiable risk factors as treatment targets. For example, hallucinations regarding self-harm would be both an indicator of risk and a target for treatment.

Implementing predictions in practice

The above-described differences in inputs and process have practical consequences for implementation. Statistical prediction occurs outside of a clinical encounter using data readily accessible to statistical models. Clinician prediction occurs in real time during face-to-face interactions. Statistical prediction is performed in large batches, with accompanying economies of scale. Clinician prediction is, by definition, a hand-crafted activity. Detailed clinician assessment may not be feasible in primary care or other settings where clinician time or specialty expertise are limited. In contrast, statistical prediction may not be feasible in settings without access to comprehensive electronic records.

The boundary between statistical prediction and clinician prediction is not crisply defined. Statistical prediction models may consider subjective information extracted from clinical text³⁷. Clinician prediction may include use of risk scores calculated from structured assessments or checklists.⁶ Additionally, clinicians determine what information is entered into medical records and thus available for statistical predictions. Despite this imprecise boundary, we believe the distinction between statistical and clinician prediction has both practical and conceptual importance.

Empirical Evidence Regarding Accuracy and Generalizability of Clinician and Statistical Prediction

Little systematic information is available regarding the accuracy of clinicians’ predictions in everyday practice. Outside of research settings, clinicians’ assessments are not typically recorded in any form allowing formal assessment of accuracy or comparison to a statistical prediction. Nock and colleagues³⁸ reported that psychiatric emergency department clinicians’ predictions were not significantly associated with probability of repeat suicidal behavior among patients presenting after a suicide attempt. A meta-analysis by Franklin and colleagues found that commonly considered risk factors (e.g., mental health diagnoses, medical illness, prior suicidal behavior) have only modest associations with risk of suicide attempt or suicide death⁵. That finding, however, may not apply to clinicians’ assessments using all data available during a clinical encounter. Any advantage of clinician prediction probably derives from use of richer information (e.g., facial expression, tone of voice) and from clinicians’ ability to assess risk in novel individual situations. No published data examine accuracy of real-world clinicians’ prediction in terms of sensitivity, positive predictive value, or overall accuracy (Area Under the Curve or AUC).

Some systematic data are available regarding accuracy of standard questionnaires or structured clinician assessments. For example, outpatients reporting thoughts of death or self-harm “nearly every day” through the PHQ-9 depression questionnaire³⁹ were eight to ten times more likely to attempt or die by suicide in the following 30 days than those reporting such thoughts “not at all”³¹. Among Veterans Health Administration outpatients, the corresponding odds ratio was approximately three⁴⁰. Nevertheless, this screening measure has significant shortcomings in both sensitivity and positive predictive value. More than one-third of suicidal behavior within 30 days of completing a PHQ-9 questionnaire occur among those reporting suicidal ideation “not at all”³¹. One-year risk of suicide attempt among those reporting suicidal ideation “nearly every day” was only four percent. The Columbia Suicide Severity Rating Scale has been reported to predict suicide attempts among outpatients receiving mental health treatment, when administered either by clinician interview or electronic self-report^{20, 21}. The Columbia scale has shown overall accuracy (AUC) of approximately 80% in predicting future suicide attempts among US veterans entering mental health care⁴¹ and people seen in emergency departments for self-harm⁴².

Statistical models developed for a range of clinical situations (active duty soldiers^{9, 43}, large integrated health systems¹¹, academic health systems^{7, 44}, emergency departments¹², and the Veterans Health Administration⁸) all appear substantially more accurate than predictions based on clinical risk factors or structured questionnaires. Overall classification accuracy as measured by AUC typically exceeds 80% for statistical prediction of suicide attempt or suicide death, clearly exceeding the accuracy rates of 55% to 60% for predictions using clinical risk factors. Overall accuracy of statistical models^{7–9, 11} also exceeds that reported for self-report questionnaires^{31, 40} or structured assessments^{20, 21, 41, 42}. As an indicator of practical utility, nearly half of suicide attempts and suicide deaths occur in patients or visits with computed risk scores in the highest 5%^{7–9, 11}.

Comparison of overall accuracy across samples (clinician prediction assessed in one sample, statistical prediction in a different sample), however, cannot inform decisions about how the two methods can be combined to facilitate suicide prevention. The future of suicide risk assessment will likely involve some combination of statistical and clinician prediction⁴⁵. Rather than framing a competition between the two methods, we should determine the optimal form of collaboration. Informing that collaboration requires detailed knowledge regarding how alternative methods agree or disagree.

Few data are available from direct comparisons of statistical predictions and clinician assessments within the same patient sample. Among ambulatory and emergency department patients undergoing suicide risk assessment, a statistical prediction from health records was substantially more accurate than clinicians’ assessments using an 18-item checklist¹⁰. Of patients identified as high risk by the Veterans Health Administration prediction model, the proportion “flagged” as high risk by treating clinicians ranged from approximately one-fifth for those with statistical predictions above the 99.9^th percentile to approximately one-tenth for those with statistical predictions above the 99^th percentile⁴⁶. In outpatients treated in seven large health systems, adding item 9 of the PHQ-9 to statistical prediction models had no effect for mental health specialty patients and only a slight effect for primary care patients⁴⁷.

While these data suggest substantial discordance between statistical predictions and clinician predictions, they do not identify patient, provider, or health system characteristics associated with disagreement. Neither do available data allow us to examine how statistical and clinician prediction agree using higher or lower risk thresholds or over shorter versus longer time periods.

Theoretically, statistical prediction can consider information available within and beyond EHRs. In statistical prediction models reported to date ^7–11, however, the strongest predictors remain mental health diagnoses, mental health treatments, and record of previous self-harm. Thus, statistical predictions still rely primarily on data created and recorded by human clinicians. Any advantage of statistical prediction derives not from access to unique data but from rapid complex calculation not possible for human clinicians.

Prediction models based on EHR data could replicate or institutionalize bias or inequity in healthcare delivery⁴⁸. Given large racial and ethnic differences in suicide mortality⁴⁹, a statistical model considering race and ethnicity would yield lower estimates of risk for some traditionally underserved⁴⁷. A model not allowed to consider race would yield less accurate predictions. If and how suicide risk predictions should consider race and ethnicity is a complex question. But the role of race and ethnicity in statistical predictions is at least subject to inspection, while biases of individual clinicians cannot be directly examined or easily remedied.

Any prediction involves applying knowledge based on prior experience. For either statistical or clinician predictions to support clinical decisions, knowledge regarding risk of suicidal behavior developed in one place at one time should still serve when transported to a later time, a different clinical setting, or a different patient population. Published data regarding transportability or generalizability, however, are sparse. Understanding the logistical and conceptual distinctions between clinician and statistical prediction, we can identify specific concerns regarding generalizability for each method. Because clinician prediction depends on the skill and judgment of human clinicians to assess risk, we must consider how clinicians’ abilities might vary across practice settings and how clinicians’ decisions might be influenced by the practice environment. Because statistical prediction usually depends on data elements from EHRs to represent underlying risk states, we must consider how differences in clinical practice, documentation practices, or data systems would affect how specific EHR data elements relate to actual risk⁴⁷. Clinical practice or documentation could vary between health care settings or within one health care setting over time⁵⁰. Even if statistical methods apply equally well across clinical settings, the resulting models may differ in predictors selected and weights applied⁵¹.

Understanding Disagreement Between Statistical and Clinician Prediction

Current¹³ and planned implementations of statistical prediction models have typically called for a two-step process with statistical and clinician prediction operating in series. That scheme is shown in the left portion of Figure 1, simplifying risk as a dichotomous state. Statistical predictions are delivered to human clinicians who then evaluate risk and initiate appropriate interventions. This two-step process may or may not be reasonable, depending on how we understand the relationship between statistical and clinician risk predictions. Those two processes may act to estimate the same or different risk states or estimands. Understanding similarities and differences in the risk states or estimands of statistical and clinician prediction is essential to optimally combining these potentially complementary tools.

Figure 1.

Alternative logical combinations of statistical and clinical prediction. In a series arrangement, clinical assessment would focus on those already identified as at risk by a statistical model. Any subsequent intervention would be limited to those considered high risk by both statistical prediction and clinical assessment. In a parallel arrangement, clinical assessment would occur regardless of statistical prediction. Any subsequent intervention would be provided to those considered high risk by either statistical or clinical prediction.

If statistical and clinician prediction are imperfect measures of the same underlying risk state or estimand, then applying these tools in series may be appropriate. Even if clinician prediction is less accurate on average than statistical prediction, combining the two may improve overall accuracy. Statistical predictions can be calculated efficiently at large scale, while clinician prediction at the level of individual encounters is more resource intensive. Consequently, it would be practical to compute statistical predictions for an entire population and reserve detailed clinician assessment and clinician prediction for those above a specific statistical threshold (Figure 1 left). Such an approach would be especially desirable if clinician prediction were more accurate for patients in the upper portion of the risk distribution (i.e. those identified by a statistical model as needing further clinician assessment).

We can, however, identify a range of scenarios when a series arrangement would not be optimal. Statistical and clinician prediction could identify different underlying risk states or estimands. For example, statistical models might be more accurate for identifying sustained or longer-term risk, while clinician assessment could be more accurate for identifying short-term or imminent risk. Alternatively, statistical and clinician predictions could also identify distinct pathways to the same outcome. For example, clinician prediction might better identify unplanned suicide attempts while statistical prediction might identify suicidal behavior following a sustained period of suicidal ideation. Finally, statistical and clinician prediction might identify risk in different populations or subgroups. For example, statistical models could be more accurate for identifying risk in those with known mental disorder, while clinician assessment could be more accurate for identifying risk among those without mental health diagnosis. In any of these scenarios, we should implement statistical and clinician prediction in parallel rather than in series. That parallel logic is illustrated in the right portion of Figure 1. We would not limit clinician assessment to those identified by a statistical model or allow clinician assessment to override an alert based on a statistical model. Instead, we would consider either as an indicator of risk.

Some more complex combination of statistical and clinician prediction methods might lead to optimal prediction of risk. Logical combination of two imperfect measures may improve both accuracy and efficiency⁵². Clinician assessment might be guided or informed by results of statistical predictions, with tailoring of clinician assessment based on patterns detected by statistical models. For example, clinicians might employ specific risk assessment tools for patients identified because of substance use disorder and different tools for patients identified because of prior suicide attempt.

Effective prevention requires more than accurate risk predictions. Both statistical and clinician prediction aim to guide the delivery of preventive interventions, most likely delivered by treating clinicians. The discussion above presumes that clinician and statistical prediction aim to inform similar types of preventive interventions. But just as clinician and statistical predictions may identify different types of risk, they may be better suited to inform different types of preventive interventions. However, for both clinician and statistical predictions, we again caution against confusing predictive relationships with causal processes or intervention targets. If recent benzodiazepine use is selected as an influential statistical predictor, that does not imply that benzodiazepines causes risk or that interventions focused on benzodiazepine use would reduce risk. Clinicians may be more able to identify unmet treatment needs than are statistical models, but correlation does not equal causation (or imply treatment effectiveness) for either clinician or statistical predictions. Caution is warranted regarding causal inference even for risk factors typically considered to be modifiable, such as alcohol use or severity of anxiety symptoms.

Priorities for Future Research

Rather than framing a competition between statistical and clinician predictions of suicide risk, future research should address how these approaches can be combined. At this time, we cannot distinguish between the scenarios described above in order to rationally combine statistical and clinician prediction. We can, however, identify several specific questions that should be addressed by future research.

Addressing any of these questions, however, will require accurate data regarding assessment of risk by real-world clinicians in actual practice. Creating those data would depend on routine documentation of clinicians’ judgements based on all data at hand. This could be accomplished through use of some standard scale allowing for individual clinician judgment in integrating all available information. The widely used Clinical Global Impressions scale and Global Assessment of Functioning scales are examples of such clinician-rated standard measures of symptom severity and disability. Clearer documentation regarding clinicians’ assessment of risk would seem an essential component of high-quality care for people at risk for suicidal behavior, but changing documentation standards would likely require action from health systems and or payers. In addition, data regarding clinicians’ assessment of risk should be linked with accurate and complete data regarding subsequent suicide attempts and suicide deaths. Systematic identification of suicidal behavior outcomes is essential for effective care delivery and quality improvement, independent of any value for research. With an adequate data infrastructure in place, we can then address specific questions regarding the relationship between statistical and clinician predictions of suicidal behavior.

First, we must quantify agreement and disagreement between statistical and clinician predictions using a range of risk thresholds. Addressing this question would require linking data from clinicians’ assessments to statistical predictions using data available prior to that clinical encounter. Any quantification of agreement should include the same at-risk population and should consider the same outcome definition(s) and outcome period(s). Ideally, quantification of agreement should consider risk predictions at the encounter level (how methods agree in identifying high-risk patients during a health care visit) and at the patient level (how methods agree in identifying high-risk patients in a defined population).

Second, after identifying overall discordance between predictions, we should examine actual rates of suicidal behavior in those who are concordant (i.e., high or low risk by both measures) and discordant (high statistical risk but not high risk by clinician assessment OR high risk by clinician assessment but not high statistical risk). Addressing this question would require linking data regarding both statistical and clinician prediction to population-based data regarding subsequent suicide attempt or death. If these two methods are imperfect measures of the same risk state, then we would expect risk in both discordant groups to be lower than that in concordant groups. Equal or higher risk in either discordant group would suggest that one measure identifies a risk state not identified by the alternative measure.

Third, we should quantify disagreement and compare performance within distinct vulnerable subgroups: those with specific diagnoses (e.g., psychotic disorders, substance use disorders), groups known to be at high risk (e.g., older men, Native Americans), or groups for whom statistical prediction models do not perform as well (e.g., people with minimal or no history of mental health diagnosis or treatment). Either statistical prediction or clinician prediction may be superior in any specific subgroup.

Fourth, we should examine how predictions operate in series (i.e., how one prediction method adds meaningfully to the other). As discussed above, practical considerations argue for statistical prediction to precede clinician assessment. That sequence presumes that clinician assessment adds value to predictions identified by a statistical model. Previous research regarding accuracy of clinician prediction has typically examined overall accuracy and not accuracy conditional on a prior statistical prediction. Relevant questions include: Does clinician assessment improve risk stratification among those identified as high-risk by a statistical model? How often does clinician assessment identify risk not identified by a statistical model? Is clinician prediction of risk more or less accurate in subgroups with specific patterns of empirically derived predictors (e.g., young people with a recent diagnosis of psychotic disorder).

Finally, we should examine the effects of specific interventions among those identified by clinician and statistical predictions of risk. Clinical trials demonstrating the risk-reducing effects of either psychosocial or pharmacologic interventions have typically included participants identified by clinicians or selected according to clinical risk factors. We should not presume that those interventions would prove equally effective in people identified by statistical predictions. Addressing this question would likely require large pragmatic trials, not limited to those who volunteer to receive suicide prevention services. If statistical prediction identifies different types or pathways of risk, people identified by statistical models might experience different benefits or harms from treatments developed and tested in clinically identified populations.

Conclusions

We anticipate both that statistical prediction tools will see increasing use and that clinician prediction will continue to improve with development of more efficient and accurate assessment tools. Consequently, the future of suicide risk prediction will likely involve some combination of these two methods. Rational combination of traditional clinician assessment and new statistical tools will require both clear understanding of the potential strengths and weaknesses of these alternative methods and empirical evidence regarding how those strengths and weaknesses affect clinical utility across a range of patient populations and care settings.