Predicting Psychological State Among Chinese Undergraduate Students in the COVID-19 Epidemic: A Longitudinal Study Using a Machine Learning

Abstract

Background

The outbreak of the 2019 novel coronavirus disease (COVID-19) not only caused physical abnormalities, but also caused psychological distress, especially for undergraduate students who are facing the pressure of academic study and work. We aimed to explore the prevalence rate of probable anxiety and probable insomnia and to find the risk factors among a longitudinal study of undergraduate students using the approach of machine learning.

Methods

The baseline data (T1) were collected from freshmen who underwent psychological evaluation at two months after entering the university. At T2 stage (February 10th to 13th, 2020), we used a convenience cluster sampling to assess psychological state (probable anxiety was assessed by general anxiety disorder-7 and probable insomnia was assessed by insomnia severity index-7) based on a web survey. We integrated information attained at T1 stage to predict probable anxiety and probable insomnia at T2 stage using a machine learning algorithm (XGBoost).

Results

Finally, we included 2009 students (response rate: 80.36%). The prevalence rate of probable anxiety and probable insomnia was 12.49% and 16.87%, respectively. The XGBoost algorithm predicted 1954 out of 2009 students (translated into 97.3% accuracy) and 1932 out of 2009 students (translated into 96.2% accuracy) who suffered anxiety and insomnia symptoms, respectively. The most relevant variables in predicting probable anxiety included romantic relationship, suicidal ideation, sleep symptoms, and a history of anxiety symptoms. The most relevant variables in predicting probable insomnia included aggression, psychotic experiences, suicidal ideation, and romantic relationship.

Conclusion

Risks for probable anxiety and probable insomnia among undergraduate students can be identified at an individual level by baseline data. Thus, timely psychological intervention for anxiety and insomnia symptoms among undergraduate students is needed considering the above factors.

Introduction

The 2019 novel coronavirus disease (COVID-19) is caused by a variety of coronavirus (SARS-CoV-2). In March 2020, the World Health Organization (WHO) declared COVID-19 as a global pandemic.¹ The symptoms of COVID-19 are usually non-specific (eg, fever, cough, and dyspnea).² Up to now, COVID-19 is significantly contagious and no effective treatments or vaccines are available.³ As of 19 June 2020 (10:00 am CET), over 8.45 million cases have been diagnosed globally with more than 453,000 fatalities.⁴ Many unprecedented strategies have been taken to cut off the spread of the virus in many countries (eg, China, England, and the United States). For example, the Chinese government released some guidance for the general population to self-isolate. Specifically, lockdown (eg, in Wuhan), temporary closure of schools/factories, and restriction of individuals’ activities

Isolation and quarantine measures have been effective at preventing the spread of COVID-19. However, consensus has arisen about their potential mental health.^5,6 In the severe acute respiratory syndrome epidemic, confirmed cases suffered higher stress levels, poor sleep quality, and depressed mood.⁷ Recently, Brooks‘ review suggested that the mental health impact of quarantine is wide-ranging and substantial.⁵ In the present epidemic of COVID-19, higher anxiety level (23.04%) was reported among medical staff.⁸ And Zhang‘s study found that more than one-third of medical staff suffered insomnia symptoms.⁹ In addition to special groups (eg, medical staff and confirmed cases), Wang’s study found that about 33.33% of the general population reported moderate-to-severe anxiety.¹⁰ Isolated people may experience stressful conditions (eg, social activities and face to face communication were restricted).¹¹ Thus, negative emotions experienced by people may be compounded.^11,12 Undergraduate education is a special stage that is highly specialized, knowledgeable, and continuous. Theory teaching and practical teaching are two profiles as well as two taches in the same teaching life-cycle. The epidemic prevented undergraduate students from returning to university. Thus, undergraduate students’ psychological state may fluctuate.^13,14

We found that previous studies have some limitations. Firstly, most research focuses on the mental health of medical students, and ignores the mental health of students of other disciplines (eg, humanities, engineering, and agricultural). Although medical students are a special group, they only constitute a very small part of undergraduate students. Secondly, most of the studies have a cross-sectional design, only exploring the psychological state and risk factors of undergraduate students during the COVID-19 epidemic.

In summary, we aimed to investigate the prevalence rate of probable anxiety and probable insomnia and to confirm the risk factors among undergraduate students during the COVID-19 outbreak. Finally, we ranked risk factors in the model based on ”feature importance.”

Methods

Participants and Procedure

We have obtained the Ethics Committee of the Ocean University of China (2,020,001). An online version of an informed consent form is provided to students before starting the survey. In the form, we explained to the students that participation was voluntary, and refusals would have no negative consequences. We also guaranteed data confidentiality and that only the researchers could access the information. Undergraduate students can choose to participate in or reject the survey. If they choose to participate in the survey, they are evaluated via an online platform. If they refuse to take part in the survey, they withdraw from the online platform. We have obtained informed consent from students who accomplished the survey. The target population of the research was undergraduate students at the Ocean University of China. The Ocean University of China is a government university located in Shandong Province. The baseline survey (T1) was collected from freshmen. Specifically, all freshmen underwent psychological evaluation at two months after entering the Ocean University of China. In the presence of the COVID-19 epidemic (T2, February 10th to 13th, 2020), we used convenience cluster sampling and invited undergraduate students (n=2500) from four grades (freshman, sophomore, junior and senior) to participate in the survey. Finally, 2009 participants completed the web-based survey (response rate: 80.36%). We used the students' ID numbers to match the data.

Measures

Social Demographic Information

Basic demographic information was collected at T1 stage. We collected basic characteristics using a self-constructed questionnaire. The questionnaire included gender (female=1, male=2), year of education (first=1, second=2, third=3, fourth=4), family economic (low family economic=1, high family economic=2), upbringing place (metropolitan=1, medium and small cities=2, town=3, country=4) and single-child families (yes=1, no=2).

College Students Mental Health Screening Scale

The Ministry of Education of the People’s Republic of China recommended the College Students Mental Health Screening Scale (CSMHSS) as a reliable and valid measurement tool to evaluate the mental health of undergraduate students. The scale of CSMHSS consists of 22 dimensions. Specifically, it includes psychotic experiences (4 items), suicidal ideation (3 items), the history of anxiety symptoms (4 items), the history of depression symptoms (5 items), paranoia (4 items), inferiority (5 items), sensitivity (4 items), social phobia (4 items), somatization (4 items), dependence (4 items), aggression (4 items), impulsive (4 items), obsession and compulsion (4 items), Internet addiction (5 items), self-injury (4 items), eating problems (4 items), sleeping problem (4 items), school adjustment difficulties (4 items), interpersonal distress (4 items), academic pressure (4 items), employment pressure (4 items) and romantic relationship problems (4 items). The scale uses 4-point Likert-scaled items ranging from 1 (not at all like me) to 4 (very like me). The scores in each dimension are added then standardized. The standard score of the total score in each dimension can be categorized at a cutoff of 3 (Fang, Yuan, Hu, Deng, and Lin, 2018).

General Anxiety Disorder-7

The scale of General Anxiety Disorder (GAD-7) was used in this research. It was a tool to assess anxiety symptoms. A score ≥7 indicated clinically significant anxiety symptoms.^15,16 GAD-7 was well validated and sensitive to the general population.^17–20 Studies had shown that GAD-7 has good reliability and validity in China.^21,22 The Cronbach’s alpha value was 0.90 for GAD-7 in this research.

Insomnia Severity Index-7

We used the Insomnia Severity Index (ISI-7) which is a 7‐item instrument that evaluates subjective sleep symptoms. Each item is scored on a 5‐point scale, with higher scores representing more severe insomnia symptoms. A score >14 indicates clinically significant insomnia symptoms.²³ The Cronbach’s alpha value was 0.86 for ISI-7 in this research.

Statistics Analysis

Descriptive data analysis was implemented in SPSS 21.0 for windows. XGBoost (Extreme Gradient Boosting), which is a machine learning algorithm, was implemented in Python 3.70. XGBoost is a method for regression and classification problems according to the Gradient Boosting Decision Tree. This method has been widely used in all kinds of data fields for regression and classification.²⁴ The algorithm of XGBoost can utilize a cross-validation approach to divide data into a model “training set” and “testing set.” In the current research, we used a 5-fold cross-validation method. Classification performance was scored with the area under the receiver-operation Curve (AUC), sensitivity (Sen), specificity (Spe), and accuracy (ACC).

Results

Participant Characteristics

Finally, there were 2009 undergraduate students who were included in our research. Of the 2009 participants, 50.97% were female, 79.99% were from high family economic background, and 25.35% were from metropolitan areas. In the present epidemic of COVID-19, the prevalence rate of probable anxiety and probable insomnia symptoms was 12.49% (GAD-7≥7) and 16.87% (ISI-7>14), respectively. The detailed basic characteristics are shown in Table 1.

Table 1.

Basic Characteristics of Undergraduate Students

		Anxiety		Insomnia
		No	Yes	No	Yes
Gender	Female	888	136	836	188
	Male	870	115	834	151
Year of education	First year	681	67	632	116
	Second year	407	66	380	93
	Third year	380	60	375	65
	Fourth year	290	58	283	65
Family economic background	Low	347	55	324	78
	High	1411	196	1346	261
Upbringing place	Metropolitan	463	69	430	102
	Medium and small cities	432	58	413	77
	Town	417	56	396	77
	Country	446	68	431	83
Only child	Yes	832	122	787	167
	No	926	129	883	172
Psychotic experiences	Yes	38	13	34	17
	No	1720	238	1636	322
Suicide ideation	Yes	28	15	23	20
	No	1730	236	1647	319
The history of anxiety symptoms	Yes	222	66	208	80
	No	1536	185	1462	259
The history of depression symptoms	Yes	495	102	463	134
	No	1263	149	1207	205
Social paranoia	Yes	134	39	119	54
	No	1624	212	1551	285
Inferiority	Yes	530	103	500	133
	No	1228	148	1170	206
Sensitivity	Yes	479	109	446	142
	No	1279	142	1224	197
Social phobia	Yes	246	50	230	66
	No	1512	201	1440	273
Somatization	Yes	60	13	47	26
	No	1698	238	1623	313
Dependence	Yes	246	48	38	26
	No	1512	203	1632	313
Aggression	Yes	48	16	38	26
	No	1710	235	1632	313
Impulsive	Yes	303	59	287	75
	No	1455	192	1383	264
Obsessive compulsive	Yes	413	87	394	106
	No	1345	164	1276	233
Internet addiction	Yes	723	99	688	134
	No	1035	152	982	205
Self-injury	Yes	14	9	11	12
	No	1744	242	1659	327
Eating problem	Yes	18	7	15	10
	No	1740	244	1655	329
Sleep problem	Yes	254	52	213	93
	No	1504	199	1457	246
School adjustment difficulties	Yes	211	38	183	66
	No	1547	213	1487	273
Interpersonal distress	Yes	134	42	120	56
	No	1624	209	1550	283
Academic pressure	Yes	903	140	850	193
	No	855	111	820	146
Employment pressure	Yes	716	102	666	152
	No	1042	149	1004	187
Romantic relationship problems	Yes	86	32	82	36
	No	1672	219	1588	303

Note: Remarks: psychotic experiences included hallucinations and/or delusions.

Predicting Probable Anxiety and Probable Insomnia

We integrated the data collected at T1 stage to predict probable anxiety and probable insomnia during the COVID-19 epidemic (at T2 stage). The AUC of probable anxiety and probable insomnia is 99.00% and 98.00%, respectively. Figure 1 shows the AUCs for probable anxiety and probable insomnia.

Figure 1.

The AUCs for the prediction of probable anxiety A and insomnia B.

“Feature Importance” in Predicting Probable Anxiety and Insomnia

According to the AUCs and the confusion matrix, we calculated the sensitivity (Sen), specificity (Spe), and accuracy (ACC). The machine learning of XGBoost predicted 1954 out of 2009 as either anxiety or no-anxiety and this translated into 97.3% accuracy (97.3% sensitivity and 96.3% specificity). The machine learning of XGBoost predicted 1932 out of 2009 as either insomnia or no-insomnia and this translated into 96.2% accuracy (95.5% sensitivity and 100.0% specificity). Detailed information is demonstrated in Table 2. Feature importance assigned positive coefficients via XGBoost, indicating that an increase in probable anxiety included 1) romantic relationship problems, 2) suicidal ideation, 3) the history of anxiety symptoms, and 4) sleep symptoms. It was also indicated that an increase in probable insomnia included 1) aggression, 2) psychotic experiences, 3) suicide ideation, and 4) romantic relationship problems. The confusion matrix and “feature importance” are shown in Figures 2A and B and 3A and B, respectively.

Table 2.

Confusion Matrix of Probable Anxiety and Insomnia

	Anxiety	Insomnia
Sen	97.3%	95.5%
Spe	96.3%	100.0%
Acc	97.3%	96.2%
Feature importance	Romantic relationship problems	Aggression
	Suicidal ideation	Psychotic experiences
	Anxiety symptoms	Suicidal ideation
	Sleep symptoms	Romantic relationship problems

Figure 2.

Confusion matrix (A) anxiety; (B) insomnia.

Figure 3.

Bar graph showing “feature importance” in predicting probable anxiety and insomnia, respectively (A) anxiety; (B) insomnia.

Discussion

Prevalence of Probable Anxiety and Insomnia

In the current research, the prevalence rate of probable anxiety and probable insomnia among undergraduate students was 12.49% and 16.87%, respectively. The prevalence rate of probable anxiety is higher than the Zhang’s study (7.5%) and lower than the Cao’s study. Cao’s study found that 24.9% of medical students suffered from anxiety symptoms.²⁵ The variability of prevalence rates could be explained by medical students being a special group who face more academic and employment pressure. And previous studies found that medical students are more likely to have psychological problems.^26,27 The prevalence rate of probable insomnia is the similarity to Huang’s study (18.2%)²⁸ and lower than the 36.1% reported by Zhang’s study.⁹ This variability of prevalence rates could be explained by the participants, questionnaires, and regions.

Prediction Factors of Probable Anxiety

Most relevant variables predicting probable anxiety included romantic relationship problems, suicidal ideation, history of anxiety symptoms, and sleep symptoms. Falling in love is a universal behavior among undergraduate students. Studies indicate that youths experience romantic relationships of joy and happiness. However, a romantic relationship is not entirely a happy period of life. Bajoghli’s study found that for youths, falling in love may be also associated with anxiety symptoms.²⁹ Consistent with Asselmann’s study, we found that the history of anxiety symptoms prior to/at baseline predicted a recurrence of probable anxiety at the time of follow-up.³⁰ Narmandakh’s study found that sleep disturbance may precede anxiety symptoms. And anxiety symptoms might be prevented by alleviating sleep disturbance.³¹ Previous results suggested that the presence of “any anxiety disorder” increases the risk for suicidal ideation among the general population, even after controlling for confounding factors (Wilcox et al, 2010). In the current study, we found that suicidal ideation can be used to predict probable anxiety. The results may indicate that there is a bidirectional relationship between suicidal ideation and anxiety symptoms among youths.

Prediction Factors of Probable Insomnia

Most relevant variables predicting probable insomnia included aggression, psychotic experiences, suicidal ideation, and romantic relationship problems. Consistent with previous studies, we also found that insomnia is a consequence of psychotic symptoms.³² Recent studies demonstrate that insomnia also contributed to the development of psychotic symptoms.³³ Insomnia symptoms may be one of the top warning signs of suicide in a clinical outpatient setting.³⁴ Suh’s study also found that insomnia symptoms were related to concurrent and future ideations of suicide in a population-based longitudinal study.³⁵ And a meta-analysis showed that sleep disturbances in general, as well as insomnia individually, appear to represent a risk factor for suicidal ideation and behavior.³⁶ Namely, there may be a bidirectional association of insomnia symptoms with psychotic experiences and insomnia symptoms with suicide ideation. Falling in love is an emotional occurrence at any age, but for undergraduate students, the feelings might be overwhelming.³⁷ In addition to being a positive feeling (eg, joy and happiness), a romantic relationship may cause stress and negative effect, especially if the feeling is not reciprocal.³⁸ Kuula’s study revealed that romantic relationship is one reason for sleep disturbance in girls and may be associated with symptoms of anxiety in both boys and girls.³⁹

Clinical Implications

We have found relatively reliable and accurate predictive models during the COVID-19 epidemic. And our models provide useful information about the most relevant variables to predict probable anxiety and insomnia among college students. The stage of university education is an important period of life development, and it is very necessary to carry out psychological assessment of freshmen who have just entered the university. Intervening with students with psychological problems in a timely and effective manner would not only help them recover their mental health, but also help them adjust their state when facing emergencies. Stoessel’s study found besides being positive feelings, romantic relationships may cause stress and negative effect, especially if the love is not reciprocal.³⁸ Thus, in addition to resolving regular psychological problems, it is necessary to help college students to establish healthy romantic relationships, one of the principal developmental tasks of emerging adulthood. How to effectively organize the mental health services for those undergraduates who have present anxiety and/or insomnia symptoms due to the COVID-19 pandemic is also very important. Community-based and school-based mental health services care be combined into the national health system.⁴⁰

Strengths and Limitations

There are some strengths in this research, including 1) this is a longitudinal study and we use the data at the time of enrollment to predict college students’ anxiety and insomnia during the outbreak; 2) we integrate data from multiple dimensions; 3) we calculate models for individual classification using machine learning. However, the current research has some limitations, including 1) our participants are from a specific university located in Shandong province. And this university does not include medical students. Thus, the results cannot be generalized to all Chinese undergraduates. 2) We used self-reported questionnaires in this research, so response bias and recall bias may exist considering that undergraduate students may have underreported or overreported their anxiety and/or sleep symptoms. We took some steps to reduce this by keeping uniformity of data collection approach. It is worth mentioning that we found that romantic relationship trouble is an important factor in predicting anxiety and insomnia. 3) We used different questionnaires at T1 stage and T2 stage. Thus, it is difficult to directly compare the prevalence rate of anxiety and insomnia at two stages. 4) Temperament is stable across the lifespan and mediate adaptive functioning to some extent. And the attachment system may be activated in stressful situations. Recently, Moccia’s research found that some specific affective temperament (eg, cyclothymic and anxious temperaments) and attachment features (eg, need for approval) can be used to predict the burden of mental health.⁴¹ However, information on temperamental and attachment was not collected in our study. Thus, it is necessary for researchers to consider temperament and attachment in future studies.

Conclusion

This longitudinal research contributes to our understanding of the psychological state of undergraduate students who suffered a sudden public health event. And we found a reliable model to predict anxiety and insomnia during the sudden public health. Thus, timely psychological intervention is necessary, not only to help undergraduate students recover their mental health but also to help them face some emergency events.

Disclosure

Fenfen Ge and Di Zhang are co-first authors. The authors report no conflicts of interest in this work.