A pilot predictive model based on COVID-19 data to assess suicidal ideation indirectly

Abstract

The COVID-19 pandemic has had a negative impact on the mental health of the population. Many studies reported high levels of psychological distress and rising rates of suicidal ideation (SI). Data on a range of psychometric scales from 1790 respondents were collected in Slovenia through an online survey between July 2020 and January 2021. As a worrying percentage (9.7%) of respondents reported having SI within the last month, the goal of this study was to estimate the presence of SI, as indicated by the Suicidal Ideation Attributes Scale (SIDAS). The estimation was based on the change of habits, demographic features, strategies for coping with stress, and satisfaction with three most important aspects of life (relationships, finances, and housing). This could both help recognize the telltale factors indicative of SI and potentially identify people at risk. The factors were specifically selected to be discreet about suicide, likely sacrificing some accuracy in return. We tried four machine learning algorithms: binary logistic regression, random forest, XGBoost, and support vector machines. Logistic regression, random forest, and XGBoost models achieved comparable performance with the highest area under the receiver operating characteristic curve of 0.83 on previously unseen data. We found an association between various subscales of Brief-COPE and SI; Self-Blame was especially indicative of the presence of SI, followed by increase in Substance Use, low Positive Reframing, Behavioral Disengagement, dissatisfaction with relationships and lower age. The results showed that the presence of SI can be estimated with reasonable specificity and sensitivity based on the proposed indicators. This suggests that the indicators we examined have a potential to be developed into a quick screening tool that would assess suicidality indirectly, without unnecessary exposure to direct questions on suicidality. As with any screening tool, subjects identified as being at risk, should be further clinically examined.

1. Introduction

The coronavirus disease 2019 (COVID-19) pandemic has affected the physical and mental health of the population throughout the world and has led to numerous negative consequences on the economy and social system (Maalouf et al., 2021). Preventive measures such as self-isolation and quarantine have been introduced during the COVID-19 pandemic and affected the daily activities, routines, and livelihoods of people. These changes in basic living conditions may increase loneliness, harmful alcohol and drug use, and self-harm or suicidal behavior (SB). There is substantial evidence to demonstrate the deterioration of mental health among people during and after the COVID-19 pandemic compared to the pre-COVID-19 period (Breslau et al., 2021; Pierce et al., 2020; Benke et al., 2020). Several authors have already cautioned about a potential increase in suicide rates during the COVID-19 pandemic (Gunell et al., 2020; McIntyre and Lee 2020), and there have been some indications of an increase in suicidal ideation (SI) among people who have been infected by COVID-19 (Na et al., 2022). Along with the rise in depression and anxiety, several studies and a recent systematic review report the rise in suicidal attempts and deaths by suicide (Pathirathna et al., 2022; García-Iglesias et al., 2022). Psychiatric emergency admissions due to SB were higher in post-lockdown time (Ambrosetti et al., 2021).

Suicide has already been identified as an acute problem in Slovenia, which has a suicide rate above the EU average, with a suicide mortality of 17.08 per 100,000 in 2018 (Roškar & Vinke, 2020). This is especially concerning given the circumstances brought on by the COVID-19 pandemic since stressful events have been established as a risk factor for SB (Buchman-Schmitt et al., 2017). Even more concerningly, the total number of suicides rose in Slovenia in 2021. A total of 432 people committed suicide in Slovenia in 2021 (338 men and 94 women), which is 63 more than in 2020 (a 17.1% increase). Even though early assumptions of suicide trends may not be appropriate, systematic reviews throughout the world show a concerning increase in SB and even completed suicides (Sher 2020; Pathirathna et al., 2022). Therefore, screening tools and strategies to detect potential at-risk people are of utmost importance. Early identification of people at risk has already been recognized as a priority by the Ministry of Health of Slovenia and measures of development and implementation of programs to enable specialists to identify and take early action in response to SB among a range of population groups are among the important goals of the mental health section.

At the country level, unemployment has been positively associated with suicide rates (Margerison- Zilko et al., 2016). Several studies have shown that in times of economic crises, rates of SI rise (Pompili et al., 2014; Fountoulakis et al., 2014; Barr et al., 2012). However, previous research (also on Slovenian data) during the COVID-19 pandemic showed that the pandemic situation might present a specific stressful event and other factors may play an important role in SB (Rus Prelog et al., 2022).

Considering all of the above, the highest importance should be placed on screening for SI in the general population, so people at risk could be directed to seek further specialized care. Suicide is a complex phenomenon with low predictive specificity (Franklin et al., 2017), and it remains a challenge to develop accurate and effective methods of identifying those likely to attempt suicide and those who have the highest risk of death by suicide (Geulayov et al., 2019). Some short screening tools exist (Mula et al., 2016; Corson et al., 2004), but they are not situation-specific, and might also be too direct in their approach and thus inappropriate for wide screening.

While depression is a strong risk factor for suicidality, SI can also exist in the absence of depression (Omary 2021; Han et al., 2015, Jadir and Anderson-Carpenter, 2022). In addition to common triggers (financial situation etc.), several other risk factors were identified during the COVID-19 pandemic (stressors related to relationships, substance use, etc.), suggesting a need for specific screening (Raue et al., 2014; Batterham et al., 2022).

To address the expected increase in the risk of SI, the aim of this study was to build a machine learning model to estimate the risk of SI discreetly, evaluate its performance, identify factors associated with the presence of SI, and assess the possibility to use the model in a screening setting.

2. Methods

2.1. Study design and sample characteristics

The data for this study came from a population-based study that was, in turn, a part of a large international multicenter study that started in Italy during the first wave of the pandemic (Giallonardo et al., 2020; Fiorillo et al., 2020) In the population-based study, we used the study protocol questionnaire adapted for the Slovenian population (Matić et al., 2022). An online survey was implemented through a multistep procedure: (a) email invitation to healthcare professionals through their institutions, (b) social media channels (Facebook, LinkedIn) with snowball sampling strategy, (c) mailing lists of universities, and (d) other official websites or mailing lists (e.g., healthcare or welfare authorities’ websites, companies, etc.). The survey took approximately 20 min to complete. The study was approved by the Republic of Slovenia National Medical Ethics Committee under the protocol no. 0120–283/2020/7.

Data collection was performed in a nationwide community sample of the Slovenian adult population. For the analysis in the present study, we used all the responses between July 23, 2020 and December 31, 2020, a total of 1790 responses.

2.2. Measures/assessment tools

The following variables were used as attributes in the machine learning setup (see Section 2.3 below). Sociodemographic variables included in the model were age, gender, and the number of people in the household (presented in Table 1 ). In addition, an extensive battery of instruments was used in the survey. However, only those relevant to this study are presented here.

Table 1.

Demographic, lifestyle, and COVID-19-related characteristics of the sample.

Marital status	Married/Living with partner	52.6
	In a relationship and not living with partner	20.3
	Single	26.9
	Divorced	0.2
Education (distribution, 9.9% without answer)	Ph.D./Specialization	4.4
	Master's degree	27.0
	Graduate degree	29.0
	High school	29.5
	Elementary school	0.2
Number of people in the household	Mean (±SD)	3.1 (±1.5)
Has children	% Yes	16.4
Has a physical illness	% Yes	10.2
Has a mental illness	% Yes	6.8
Has tested positive for COVID-19	% Yes	7.3
Has been hospitalized for COVID-19	% Yes	4.7
Has had pneumonia caused by COVID-19	% Yes	0.0
Has been in isolation due to COVID-19	% Yes	17.5
Has been in isolation due to contact with a person infected with COVID-19	% Yes	13.6
Close person has had COVID-19	% Yes	17.9
Close person died of COVID-19	% Yes	0.3
Pregnant women or newborn in the household	% Yes	6.6
Elderly or chronically ill person in the household	% Yes	22.8

The variables related to satisfaction since the beginning of the COVID-19 pandemic were assessed with three questions regarding the perceived satisfaction with (1) finances (e.g., “Following the pandemic, how satisfied are you with your financial situation?”), (2) relationships with people they lived with during the pandemic, or relationships with their closest people if they lived alone, and (3) housing. All three variables were assessed on a 7-point Likert scale (ranging from “could not be worse” to “could not be better”). These three variables were used individually as they do not form a predefined scale.

The attributes related to changes in habits were: increase in substance use (any of the following: tobacco, alcohol, marijuana, other psychoactive substances, and drugs not prescribed by a physician), quantitative change in the level of physical activity and eating (either increase or decrease for both), and the estimated daily amount of time spent online.

Coping with stress was assessed with The Brief COPE, a 28-item multidimensional measure of strategies used for coping in response to stressors (Carver, 1997). This abbreviated inventory (based on the complete 60-item COPE Inventory) is comprised of items that assess the frequency with which a person uses different coping strategies (e.g., “I've been turning to work or other activities to take my mind off things”, “I've been making fun of the situation”, “I've been criticizing myself”) rated on a scale from 1, “I haven't been doing this at all”, to 4, “I've been doing this a lot”. There are 14 two-item subscales within the Brief COPE, and each is analyzed separately: (1) self-distraction, (2) active coping, (3) denial, (4) substance use, (5) use of emotional support, (6) use of instrumental support, (7) behavioral disengagement, (8) venting, (9) positive reframing, (10) planning, (11) humor, (12) acceptance, (13) religion, and (14) self-blame. Each subscale's score is calculated as a sum of two items.

SI (the target variable in the machine learning setup) was assessed using the Suicidal Ideation Attributes Scale (SIDAS) (van Spijker et al., 2014). The SIDAS instrument consists of five items assessing the frequency of suicidal thoughts, controllability, closeness to attempt, level of associated distress, and interference with daily functioning over the past month. Each item is assessed on a 10-level Likert scale, with a total score ranging from 0 to 50. In the case of scoring “0—Never” on the first item, all other items are skipped, and the total score is zero. The presence of any SI is considered indicative of a risk for SB (Reynolds and Fletcher-Janzen, 2004), which is why we chose to divide our sample into those scoring zero, who record an absence of SI, and those who score one or higher, who record the presence of SI in the past month. Among the 1790 respondents, 173 (9.7%) scored above zero.

2.3. Machine learning setup

Machine learning methods were used to both: try to identify respondents at risk of SI, and to assess which factors are most indicative of SI. Four algorithms were used to build the classification models and their performance was compared: logistic regression (LGR) (Hosmer and Lemeshow, 2000), XGBoost (XGB) (Chen and Guestrin, 2016), random forest (RF) (Breiman, 2001), and support vector machines (SVM) (Cortes and Vapnik, 1995). Hyperparameters of all four models were tuned using a standard grid-search approach with 10-fold cross-validation to obtain the highest area under the receiver operator characteristic curve (AUROC).

The target variable for all models was the presence or absence of SI. Thus, the models were trained to detect SI presence (SIDAS score above zero) or SI absence (SIDAS score equal to zero). The attributes were as described above.

Before training the classification algorithms, 537 cases (30% of the data) were set aside to serve as a test set, using stratified sampling with respect to the SI category, so that the proportions of the target variable were kept close to those in the entire dataset (52 cases with presence of SI, or 9.7%).

As a preprocessing step, based on the 70% of data that went into the training set, all continuous attributes were scaled on a scale from 0 to 1, and a small number of missing values (18 cases) on Instrumental support subscale of Brief-COPE was replaced by the mean. No other preprocessing was performed.

The receiver operator characteristics (ROC) curve was used for the analysis of potential decision cut-offs to balance the sensitivity/specificity tradeoff.

3. Results

The results reported here were all obtained on an independent test set that was set aside exclusively for evaluation purposes. The LGR, RF, and XGB models performed more or less equally well with AUROC of 0.83, 0.82, and 0.82, respectively. The SVM model performed worse with an AUROC of 0.76. In the continuation, we report the results of the LGR model primarily due to its much better interpretability.

The factors most strongly indicating the presence of SI in the LGR model were analyzed and we report them along with their unstandardized coefficients (note that the attributes themselves were scaled during preprocessing) in decreasing order of their influence. The sign was already accounted for in the description, e.g. lower age since the sign is negative. The capitalized factors are Brief-COPE subscales (each sum of two items). The factors were: Self-Blame subscale (b = 2.05) as the most influential one, followed by increase in Substance Use (b = 1.48), low Positive Reframing (b = −1.35), Behavioral Disengagement (b = 1.23), dissatisfaction with relationships (b = −0.84), lower age (b = −0.69), dissatisfaction with housing (b = −0.61), Denial (b = 0.58), Self-Distraction (b = 0.52), change (increase) in substance use (b = −0.42), Religion (b = 0.39), Venting (b = 0.39), male gender (b = 0.35), lower Active Coping (b = −0.32), and change in physical activity (b = −0.25). The double occurrence of increase in substance use is not a mistake – one question is part of Brief-COPE and the other from the question about the change in habits.

All other factors had coefficients lower than ±0.25. These remaining factors are: Humor (b = 0.19), lower Instrumental Support (b = −0.18), decreased Planning (b = −0.17), lower Acceptance (b = −0.15), dissatisfaction with finances (b = −0.13), no change in eating habits (b = 0.10), lower number of people in the household (b = −0.09), time spent online (b = 0.08), and lower Emotional Support (b = −0.03).

The above factors, as seen from their coefficients, play a lesser role in the model, especially the ones at the very end. However, we listed them here as they complete the model. Finally, the intercept (constant) in the LGR model is −1.71.

The factors, as presented here, should not be interpreted in the sense that they are the underlying cause of SI. They are used by the model to estimate the likelihood of SI for a given case.

A more detailed understanding of how the LGR model performs on previously unseen cases (test set) can be drawn from Fig. 1 . The red bars represent respondents with nonzero scores for SIDAS and the green bars represent respondents who scored zero for SIDAS. The color of the bars thus represents the ground truth (actual response). The height of the bars conveys the probability of a respondent scoring nonzero for SIDAS that the LGR model assigned to that particular respondent based on his or her responses (attributes). For example, the rightmost red bar represents a respondent whose SIDAS score was nonzero, and the LGR model assigned this respondent the probability of a nonzero SIDAS score of only about 8% (the worst false negative case).

Fig. 1.

Performance of the LGR model on previously unseen data (test set).

The respondents are sorted by their estimated probability of scoring nonzero for SIDAS, from highest to lowest. It can be seen that the majority of red bars are clustered on the left (high probability of nonzero SIDAS) and they get ever more sparse towards the right side (lower probability of nonzero SIDAS). To use the model in practice we need to assign a threshold above which estimated probability we will consider the respondent at (high enough) risk and identify him or her as being “red”. Of course, this threshold depends on the purpose and considerations (e.g. available capacity for further diagnostic procedures) of the actual practical application of the model. One such potential threshold, set at 60%, is depicted in Fig. 1. In practice, as estimated from the test set data and seen in the figure, this would represent a situation where around 10% of the population would be assessed as “red” (those with the highest LGR assigned probabilities) and if this 10% were to be further diagnosed, we could expect to discover about 46% of all actual respondents who have a nonzero score for SIDAS.

As we briefly mentioned above, threshold selection depends on the considerations of the concrete purpose of the intended application. A selection of further interesting threshold candidates representing different situations can be seen from the ROC curve of the LGR model shown in Fig. 2 . We can see that if the goal is to detect the three-quarter majority of those at risk of SI that means scanning about a quarter of the population - which is likely infeasible. In this context, scanning means a further diagnostic activity applied to those that were identified as at risk by the model (scored above the selected threshold). On the other side of the spectrum, further diagnostic scanning of only about 3% of the target population (e.g. in a specific region), would uncover about 14% of those actually at risk. Such a yield for a relatively manageable effort may well be worth considering in many situations.

Fig. 2.

Receiver operator characteristics curve for the LGR model with some select decision threshold candidates.

4. Discussion

Nearly 10 percent (9.7%) of respondents reported SI during the past month, similar to other studies with a comparable pandemic time frame (Czeisler et al., 2021; Bersia et al., 2022). That indicated a need for a means to reach and identify the at-risk population in a widespread and indirect manner. At the same time, such a proportion of respondents reporting SI enabled machine learning methods to be used as there was enough labeled data and the two classes (SI yes and no) were not so severely unbalanced. On top of that, there was a wealth of data collected that could be used as attributes.

We employed and compared four classification algorithms in their ability to classify respondents as having or not having SI based on selected demographic attributes, satisfaction with three aspects of life, current habits and changes in habits since the beginning of the COVID-19 pandemic, and strategies for coping with stress. These attributes were specifically selected to be discreet about suicide, likely sacrificing some of the model's accuracy in return. We observed that, with exception of SVM, the algorithms yielded comparable and quite encouraging results with AUROC reaching above 80%. We selected the logistic regression model for further analysis as it is the most comprehensible among the three best-performing models.

We found an association between various subscales of Brief-COPE and SI, which is in line with previous research (Hallford et al., 2022). Self-Blame was especially indicative of the presence of SI, followed by low Positive Reframing and Behavioral Disengagement. Self-blame and behavioral disengagement coping strategies were previously found to be positively associated with past-year SI in a sample of college students (Goodwill, 2022). Negative and dysfunctional coping skills and negative emotions are known to trigger a higher risk for suicide (Okechukwu et al., 2022).

Consistent with other studies during this pandemic (Batterham et al., 2022; Bersia et al., 2022), there is a concerning trend of younger people being more likely to experience SI. As in several other countries that explored SI in the same timeframe (Cheung et al., 2021), male respondents were more likely to report SI than female ones. Most studies showed women had worse mental health outcomes during the pandemic than men (Rus Prelog et al., 2022; Almeida et al., 2020; Thibaut and van Wijngaarden-Cremers, 2020), yet men remain more at risk for suicide, which further emphasizes the need for a more specific screening.

In our study, at least in the context of the COVID-19 pandemic, dissatisfaction with close relationships and housing seem to be more important indicators of SI than dissatisfaction with the financial situation, which is explainable, given that during the lockdown measures, people have been bound to their homes and living environment for several weeks, with limited contact outside the family/closest people. Similarly, findings from a study on undergraduate students reported a strong positive association between poor housing quality and mental health outcomes, such as depressive symptoms and impulsivity, especially in males (Amerio et al., 2022). Regarding interpersonal relationships during economic crises, it had already been demonstrated, positive relationships and trust were the only statistically significant factors in mitigating mental health consequences (Economou et al., 2013). This was further substantiated and “Interpersonal Trust” theory has been reported to be relevant to profound effects on mental and physical health under the most adverse conditions as well. During the recent economic crisis in Greece, “Interpersonal Trust” emerged as the only significant protective factor with respect to SI (Economou et al., 2013). Among the factors worth considering that underlie this psychological distress, are fears related to COVID-19. Analyzed in light of age groups, a study found that in the youth group, fears were commonly associated with interpersonal factors (Costanza et al., 2021b, Costanza et al., 2021a). Finally, in line with other studies (Czeisler et al., 2021; Panchal et al., 2020), we found that a change in substance use since the start of the pandemic was associated with SI.

The results showed that the presence of SI can be detected with reasonable specificity and sensitivity based on the said indicators. The associations featured in the model, as discussed above, are consistent with many previous studies. This bodes well for the soundness of the machine-learned model. The potential tool, based on these models, would require about 5 min to administer and would not approach suicidality directly, which can be important for wider use at the primary level in clinical (and non-clinical) settings. While several tools that assess mental health and suicidality exist and are in use (Horowitz et al., 2013; Mullinax et al., 2018), many people feel reluctant to take part in such surveys and even many clinicians are reserved to the use of screening tools routinely. It is, however, important to note that tools can never replace the clinical examination; for people at risk, exploring suicidality in clinical context within a therapeutic relationship is vital.

Even though it might still be early to evaluate trends in suicidality in the post-pandemic era, suicide prevention remains the highest priority. Higher psychological distress can lead to elevations in SI during the pandemic, especially in certain groups. By identifying some specific factors that correlate with SI, targeted prevention strategies could also be formed to mitigate harm. Subtle screening tools for the general population that can be administered in clinical or specific non-clinical settings (and can be web-based to reach more people) can provide a useful solution for the detection of people at risk.

4.1. Limitations and strengths

The present study's findings must be interpreted with consideration of design limitations. First, this study is cross-sectional and, as a result, none of the associations presented should be interpreted as causal. Future longitudinal research is necessary to draw conclusions about the temporal relationship between COVID-19-related distress and suicidality.

An advantage of the pandemic timeframe of the study is that the specific pandemic situation allowed for a better observation of suicidality with sufficient SIDAS-positive individuals. A major element of novelty of our study is the potential development of a new screening questionnaire/tool, that could be broadly generalizable, would take only a few minutes to administer, and could detect the possible at-risk population that would be further examined. At least in the context of such an event as the COVID-19 pandemic, that could prove highly important.

Besides the reasonable accuracy, the following strengths are worth mentioning. A tool, based on the model, is easy to administer either online or in person and, as already mentioned, takes only minutes to complete. Another strength is the explainability of the LGR model. The factors playing a (significant) role for risk can be given per concrete subject. This is quite a standard machine learning “explanation” of why a certain case was assessed as either at risk or not. An example of such an explanation is e.g. “the subject is at risk, because he is a young male, dissatisfied with relationships, not actively coping, and with expressed change in substance use”. Such an explanation simply observes which factors significantly contributed to the total score using the known weights of the factors (listed in the results section). The reader should note, however, that while the model uses the factors to assess the likelihood of the subject being potentially at risk of SI, it does not in any way imply that these factors are the underlying cause of SI.

The findings need to be interpreted cautiously in light of these limitations because the obtained results might not be generalizable in non-pandemic situations when global stress is not present. Also, the resulting features and threshold values are specific to a particular situation. This can nevertheless suggest that these characteristics can be useful, probably not only in an epidemic situation. The next step could be to encompass correlates of depression since SI and depression often present together.

5. Conclusions

Within this study, we developed a machine learning model for detecting SI indirectly, without unnecessary exposure to direct questions on suicidality. The detection of SI was based on habits, demographic features, strategies for coping with stress, and satisfaction with three important aspects of life. A logistic regression model was selected as the best option based on its performance (AUROC = 0.83) and its comprehensibility. The model's most important attributes coincide well with the results of previous studies.

The proposed tool (classification model used in clinical decision support or screening setting) is merely a pilot at this stage. With this study, we primarily wanted to check the feasibility and approximate performance envelope of such a tool/model. The pandemic setting of the collected data should also be taken into account and it remains to be seen whether the model's accuracy holds only for the pandemic situations or it is more general.

Finally, it is important to note that the tools can never completely replace direct questions on suicidality; with people at risk, clinical evaluation is crucial.

Author statement

Polona Rus Prelog: Conceptualization, Investigation, Writing, Original draft.

Preparation. Teodora Matić: Data curation, Methodology, Visualization. Peter Pregelj: Supervision and Editing. Aleksander Sadikov: Methodology, Supervision, Editing, Writing.

Funding/acknowledgments

This research was supported by the Slovenian Research Agency (ARRS) under the Artificial Intelligence and Intelligent Systems Programme (Grant No. P2-0209).

Declaration of competing interest

None.