Online Mental Health Survey for Addressing Psychosocial Condition During the COVID-19 Pandemic in Indonesia: Instrument Evaluation

Abstract

Background

Regular monitoring of the pandemic’s psychosocial impact could be conducted among the community but is limited through online media. This study aims to evaluate the self-rating questionnaire commonly used for online monitoring of the psychosocial implications of the coronavirus disease 2019 (COVID-19) pandemic.

Methods

The data were taken from the online assessment results of two groups, with a total of 765 participants. The instruments studied were Self-Rating Questionnaire (SRQ-20), post-traumatic stress disorder (PTSD), and Center for Epidemiological Studies Depression Scale-10 (CESD-10), used in the online assessment. Data analysis used Rasch modeling and Winsteps applications. Validity and reliability were tested, and data were fit with the model, rating scale, and item fit analysis.

Results

All the scales for outfit mean square (MnSq) were very close to the ideal value of 1.0, and the Chi-square test was significant. Item reliability was greater than 0.67, item separation was greater than 3, and Cronbach’s alpha was greater than 0.60; all the instruments were considered very good. The raw variance explained by measures for the SRQ-20, PTSD, and CESD-10 was 30.7%, 41.6%, and 47.6%, respectively. The unexplained eigenvalue variances in the first contrast were 2.3, 1.6, and 2.0 for the SRQ-20, PTSD, and CESD-10, respectively. All items had positive point-measure correlations.

Conclusion

The internal consistency of all the instruments was reliable. Data were fit to the model as the items were productive for measurement and had a reasonable prediction. All the scales are functionally one-dimensional.

Introduction

The coronavirus disease 2019 (COVID-19) attacks many countries with long incubation periods, high morbidity, and mortality rates for both the community and health workers.^1,2 This pandemic had a major impact on all aspects of people’s lives and the country, both economically and public health such as international travel dropped drastically, school closure happened everywhere and remote working become common, so that monitoring and surveillance efforts were initiated.^3,4 Surveillance efforts are not only related to the disease but also its effects on psychosocial and economic problems which significantly impact to the people way of life.⁵ Government decisions made to reduce disease spread by isolation and quarantine measures change the structure of activities that are at the core of mental health, including social connectivity, daily routine activities, access to information sources, and the duration and frequency of device use.^6,7

Many monitoring and research activities have been performed on COVID-19 and its health and socio-economic effects. In the context of public health, disease surveillance, and mitigation of the impact of COVID-19 are needed to develop intervention strategies to deal with the pandemic and its effects. Monitoring and research conducted in pandemic situations face problems with direct measurements because of restrictions to reduce disease transmission, several studies show about this such as in China,^8–10 Spain,⁸ USA,⁹ Poland,¹⁰ Iran,¹¹ the Philiphines¹² and Vietnam.¹³ Those studies inform direct challenges that has never been face before in many parts of the world. Therefore, many of these activities are conducted online. This arrangement can affect the quality of the process and the results of the research. However, this method is a safe option for both public and research interests.

This public health emergency does not only affect physical health but also affects one’s mental health.¹⁴ The assessment of psychosocial status in the general population is very important to reduce the negative impact of a pandemic. Several independent research institutes and researchers measure the state of mental disorders in the community. Psychometric standard instruments are widely used and administered online.

Self-Rating Questionnaire (SRQ) is a screening instrument that is self-administered online to screen for mental health disorders in general.¹⁵ The SRQ is a screening instrument used by the Ministry of Health of the Republic of Indonesia to conduct Basic Health Research.^16,17 The measurement results with this instrument can quickly detect emotional and mental disorders.

One of the negative effects that need to be evaluated is post-traumatic stress disorder (PTSD). PTSD usually occurs after more than one month but within six months of the traumatic event and causes suffering or significant functional impairment. Traumatic stress occurs beyond the limits of ordinary human experience. This condition occurs when a person experiences serious threats to their own psychological and physical safety or their loved ones.^2,18,19 PTSD has been reported as a psychological effect of SARS and virus H1N1 epidemics, and recently used in COVID-19 pandemic studies too.^20,21 Therefore, PTSD should also receive attention in the COVID-19 pandemic. PTSD causes dramatic changes in mood and cognition. Based on the PTSD cognitive model, negative emotions (sadness, fear, anger) can cause a person to adopt negative judgments and negative coping styles.^2,6,19

The CESD (Center for Epidemiological Studies Depression Scale) is a screening tool used to detect cases of depression in the general population.^22,23 The CESD-10 is a more concise version and is effective enough to be used in surveillance by health professionals and for confirmation by specialists at a later stage. Are these instruments then reliable for measuring online? It is necessary to evaluate this method so that its sustainable use for disaster mitigation and surveillance of a pandemic’s impact can be continued. This study aims to evaluate the self-rating questionnaires commonly used for online monitoring of the psychosocial implications of the COVID-19 pandemic. The research results are expected to be useful to researchers and the public in mental health monitoring and surveillance and further online research in an epidemiological study.

Materials and Methods

Research Design

We conducted several online pandemic survey packages for different groups covering the general national population limited to specific populations. For instrument evaluation, we decided to select data from two surveys that used the same three instruments and had their data entered on approximately the same date.

There were 765 respondents who participated in this study from West Java Province, Indonesia. Convenience sampling technique was used, respondents were administered an online survey via the mailing list/online platform. With regard to ethical considerations, the participants’ consent to take part in this study was obtained before they filled in the questionnaire. On the front cover of the questionnaires’ online form, it was stated that the respondents are given a choice whether to participate in the survey. Participation was strictly voluntary and anonymous. Thus, by completing the questionnaire, the respondents provided their consent. Details of the respondent demographic profiles are shown in Table 1.

Table 1.

Demographic Data of Respondent (N = 765)

Demographics	Frequency	Percentage
Gender
Male	433	56%
Female	332	44%
Age
Less than 21 years	79	18%
21–40 years	380	49%
41–55 years	221	29%
More than 55 years	85	11%
Marital Status
Single	191	25%
Married	532	69%
Divorce	42	6%

Instrument

Three instruments were used in this study: the SRQ (Self-Rating Questionnaire), PTSD (Post-Traumatic Stress Disorders), and CESD-10 (Center for Epidemiological Studies Depression Scale). The Self Reporting Questionnaire has been developed by WHO to screen psychiatric disturbance. In Indonesia SRQ consist of 20 items (SRQ-20) was widely used in primary health center, hospitals, and Ministry of Health to screen psychiatric symptoms like neurotic, substance abuse, psychotic symptoms. Screening with SRQ-20 is widely used because easily to be used, short period of time for filling the questionnaire, self-reported with two choices (Yes or No).^24–31

The PTSD instrument is for measuring psychological situations and consists of five items with two choices (Yes or No). In clinical area, the PTSD is used to helped health professional identified emotional disturbance in daily practice. The code for PTSD is F 43.10 with consist of persistent re-experienced symptoms, persistence avoidance of stimuli associated with trauma, persistence symptoms of increased arousal which not present before the trauma with two choices (Yes or No).^32,33 Meanwhile, the CESD-10 is a screening tool for depression with a scale consisting of 10 items that provides five choices (0 to 4) each.

The Model of Measurement

Rasch model analysis was employed in this study to analyze the data collected from the three questionnaires. This study was chosen as it can provide accurate and precise latent trait measurements about measuring mental health. According to Wright and Mok, a good and valid measurement model has to follow five measurement principles for human science. They are to (a) yield a linear measure, (b) overcome missing data, (c) provide a precision estimate, (d) discover outliers or misfits, and (e) be replicable.³⁴ The Rasch model fulfills these compared with other measurement models.

The Rasch model is a subset of a larger group of measurement models, called item response theory (IRT), which transform raw ordinal type data using probability and logarithms to become equal-interval scale data called logit (log odd unit). The Rasch model had been used widely to analyze psychometric data in many fields, such as educational research, language assessment, and health sciences.^35–39 Rasch model analysis also provided an extremely effective alternative to investigate the psychometric properties of a cognitive and non-cognitive instrument to address response bias.^40,41

All collected raw data was input into a Microsoft Excel file, checked by the Rasch measurement model software WINSTEPS version 3.73, for data validation and cleaning. No missing data were found. Rasch analysis was performed using the Rasch Rating Scale Model (RSM), an extension of the Rasch model for dichotomous data developed by Andrich.³⁸

The content validity and internal consistency reliability of each scale were determined to validate the three instruments used in this study. The first stage is to identify at the instrument level, which includes data fit to the model, Chi-square test, item reliability, item separation, and Cronbach’s alpha indices were examined as well. The uni-dimensionality requirement was investigated using the Principal Components Analysis of Rasch measures and residuals. It can be asserted that the data is fundamentally one-dimensional if the Rasch measurement indicates a relatively elevated percentage of explained variance (at least 20% for dichotomous data, which were the SRQ-20 and PTSD questionnaires, and at least 40% for polytomous data applied to the CESD-10 questionnaire) and the first residual components of the unexplained variances are less than 3 eigenvalues.⁴²

Rating scale analysis is another examination to check the effectiveness rating given to respondents. Several criteria have been suggested by Linacre to diagnose a malfunctioning empirical rating scale used in the instrument.⁴³ A rating scale at an ideal point can be considered when: a) there are at least 10 observations in each category, b) the person’s measurements are an average by category and increase monotonically with the rating scale, c) outfit MnSq should be less than 1.5 in each step, and d) step difficulties should advance no less than 1.4 and not more than 5 logits.⁴⁴

The appropriateness of the item quality was checked two indices because the number of respondents was more than 500 which are the Outfit MnSq and Point-measure correlation (Pt-Measure Corr) for each item.^36,45 The Wright map was displayed to demonstrate the spread item difficulty and respondent abilities comprehensively (Figures 1,2,3).

Figure 1.

Variable map of SRQ-20.

Figure 2.

Variable map of PTSD.

Figure 3.

Variable map of CESD-R10.

The presence of test items that functioned differently, a response bias, was explored for each respondent’s demographic variable. DIF analysis was also executed. A moderate item DIF was considered to be present if the difficulty parameters between different groups of demographic variables (such as male and female) fulfilled three criteria, which are if it had a t-value of less than −2.0 or more than 2.0, DIF contrast value of less than −0.5 or more than 0.5, and the p (Probability) value of less than 0.05 or greater than −0.05.^36,37,46

Ethical Consideration

This research is in accordance with the Declaration of Helsinki and has obtained ethical clearance from the Ethics Committee of the Faculty of Medicine Universitas Padjadjaran No. 496/UN6.KEP/EC/2020. All participants gave written informed consent about the purpose of the study.

Results

Reliability and Data Fit to the Model

The content validity of the three questionnaires was identified by assessing fit validity. The Rasch instrument-fit statistics assessment was utilized to determine the degree to which data fit with the model and, therefore, fit the concept of a single attribute.³⁸ Index of Outfit MnSq as shown in Table 2, it was found that all the scales were very close to the ideal value of 1.0 and within the acceptance ranges of 0.5 to 1.5.³⁶ This result was also supported by the Chi-square test where all instrument results were significant. Hence, the data fit the model as items were productive for measurement and had a reasonable prediction value. For measuring reliability, three indices, which were item reliability (should be more than 0.67), item separation (more than 3), and Cronbach’s alpha (more than 0.67), demonstrated that all three instruments were considered very good. Therefore, the internal consistency of all instruments was shown to be reliable.

Table 2.

Reliability and Model Fit (N = 765)

Psychometric Attribute	Instrument
	SRQ-20	PTSD	CESD-10
Number of item	20	5	10
Outfit Mean Square
Mean	0.93	1.05	1.01
Standard Deviation	0.33	0.06	0.44
Separation	7.46	10.84	7.77
Reliability	0.98	0.99	0.98
Cronbach’s alpha	0.88	0.62	0.86
Chi-square (χ2)	8099*	1786*	13,221*
Unidimensionality
Raw variance	30.7%	41.6%	47.6%
Unexplained variance 1st contrast	2.3	1.6	2.0

Note: *p < 0.01.

The analysis of dimensionality in the Rasch model showed that three of the scales met the assumptions of unidimensionality, both for the dichotomous data scale (SRQ-20 and PTSD) and polytomous data of CESD-10. The raw variance explained by measures for the SRQ-20, PTSD, and CESD-10 was 30.7%, 41.6%, and 47.6%, respectively. The unexplained eigenvalue variances in the first contrast were 2.3, 1.6, and 2.0, for the SRQ-20, PTSD, and CESD-10, respectively. These results show that all the scales are functionally one-dimensional.

Rating Scale Analysis

The effectiveness of the rating scale used in each questionnaire was examined. Results illustrated that there were no disordered thresholds from one scale to another (Table 3). All categories in the three questionnaires were chosen by more than 100 respondents. The average person measured by category moved up monotonically with the respective rating scale for the SRQ-20, PTSD, and CESD-10 scales. All categories had outfit mean squares close to the ideal value of 1.0 in each step, except for rating 4 on the CESD-10 scale that had a value of more than 1.5. Gaps in step difficulties are always obvious for dichotomous data, in this case, the SRQ-20 and PTSD scales.^36,38 Also, for the CESD-10 scale with polytomous data (5 ratings), the steps between rating scales are greater than 1.4 and lower than 5 logits.⁴⁴ Hence, all Likert scale categories in the three questionnaires used in this study were well functioning.

Table 3.

Rating Scale Model Category Statistics (N = 765)

Instrument	Category	Frequency	Percentage	Average Measure	Outfit MnSq	Step
SRQ-20	1 (Y)	12,759	83%	−2.27	1.01	-
	2 (N)	2541	17%	−0.12	0.90	-
PTSD	1 (Y)	2983	78%	−1.90	1.06	-
	2 (N)	842	22%	0.77	1.05	-
CESD-10	0 ()	403	5%	−3.92	0.90	None
	1 ()	3213	42%	−1.40	0.92	−4.58
	2 ()	3114	41%	0.15	0.92	−0.55
	3 ()	752	10%	1.14	1.10	2.05
	4 ()	168	2%	1.57	1.57	3.08

Item Fit Statistics Analysis

The fit of the data to the Rasch model was analyzed for all items in each scale in the study. The mean of item measure (logit) of three questionnaires has an ideal value of 0.0 logit, showing that all scales have an ideal starting point for measurement (Table 3), and none of the three scales has an extreme measure value. The values of the standard deviation for two scales have greater (more than 1 logit) values. This means that they are better regarding the spread of item difficulty, which was in the SRQ-20 and PTSD questionnaires; for the CESD-10 scale, the standard deviation is half (0.51 logits). This also can be seen in the range of item measures on each scale, whereas CESD-10 has less range than SRQ-20 and PTSD scale. There were three items, two in the SRQ and one in the CESD-10 scale, that were considered misfit item values of the outfit MnSq larger than 1.5. However, all items had positive point-measure correlations (there was no item has polarization), the range was greater than 0.4 for PTSD and the CESD-10 scale.

The DIF analysis was conducted to examine the item bias according to each demographic variable collected, which were gender, age, and marital status of the respondent, on each scale. As shown in Table 4, item DIF is nonexistent only for male and female respondents in the CESD-10 scale; it has item DIF for another situation. The lowest number was on the PTSD scale (three items), followed by the CESD-10 scale (4 items) and the SRQ-20 scale (15 items).

Table 4.

Item Fit Statistics (N = 765)

Psychometric Attribute	Instrument
	SRQ-20	PTSD	CESD-10
Number of items	20	5	10
Item Measure
Mean	0.00	0.00	0.00
Standard Deviation	1.23	1.74	0.51
Range of item measure	–1.90 to 3.59	–1.88 to 2.94	–0.85 to 0.79
Extreme measure	None	None	None
Point measure correlation	0.34–0.66	0.41–0.76	0.46–0.76
Misfit item (Outfit MnSq > 1.5)	2 items	None	1 item
Item DIF based on demographic
Gender, item no:	7, 10, 11, 13, 14, 18, 19, 20	2, 3	None
Age, item no:	1, 3, 5, 7, 11, 12, 13, 18, 19, 20	5	5, 9
Marital Status, item no:	1, 6, 7, 10, 12, 13, 15, 16, 17, 18, 19	5	1, 5, 7, 9

Discussion

Measurements require a valid and reliable instrument so that they can be used in epidemiological studies. Evaluation of the three instruments show that the SRQ-20, PTSD, and CESD-10 are valid and reliable to be used as online measuring tools. In practice, health office, surveillance officers on the front line or at institutions who intend to monitor their staff’s psychological impact can use these instruments without hesitation.

We reported items showing misfit: 2 items on the SRQ-20 and 1 item on the CESD-10. When an item is found that misfits the model, it means that it is not working as expected in the model. Misfit can occur and be caused by the participants’ interpretation of the item statement or treatment of online instruments. This can happen because of the use of online instruments.

The treatment of the misfit item is to remove the item. However, this will interfere with the measurement results and is very detrimental to the measurement. These instruments have also been used in daily practice for a long time and may not align with these instruments’ developer. Is the measurement result with the misfit item still acceptable? Kohler and Hartig suggest comparing the results with inclusion and those with the deletion of misfit items. If there is a decrease in reliability, the items that are misfit are included.⁴⁷ Thus, the instruments used online should be evaluated and conveyed in each measurement result.

Instrument users also need to pay attention to the occurrence of DIF in certain groups. DIF means that there is a group influence on the measurement results. DIF does not mean merely measurement bias. The DIF analysis results provide support for interpreting research results with awareness in the group where DIF occurs. The Rasch model is more convenient than other, more complex models for detecting DIF because it can determine the whole group’s item parameters without the influence of confounding parameters.⁴⁸ DIF contributes to better accuracy of the final results in epidemiological studies because it increases the scale of the composite measurement.⁴⁸

COVID-19 is a disease with a high infection rate and a long incubation period caused by SARS-CoV-2.⁴⁹ It has had a significant effect on the global economy and public health. Although patients and the community have been affected, mental health is not prioritized, even though it imposes a formidable burden.⁶ In the early days of the pandemic, large parts of society experienced much fear and anxiety.^50,51 Several cases of suicide and attempted suicide in Indonesia occurred in patients who were treated and doing self-isolation.⁵²

Social restrictions are executed in the form of wearing face mask, lockdowns, quarantines, or social restrictions.^10,53 This disruption creates mental pressure on individuals. These restrictions also affect economic activity, cause unemployment and lack of income, and disrupt food access, thereby ultimately causing anxiety, fear, and panic.^50,54 Restricted religious activities create a religious sentiment, tension, and collective pressure in society. The implementation of the quarantine and isolation policy (lockdown) and the increasing number of cases affect the core of human mental health, including relationships in daily routine activities and adverse psychosocial effects.^6,19

Studies showed 42.65% and 67.09% of participants self-reported psychiatric disorders and a high level of PTSD, respectively. At the initial phase of the disease, there was a high prevalence of depression (48.3%), anxiety (22.6%), and a combination of depression and anxiety (19.4%). The study results of COVID-19 survivors indicated that 28% for PTSD, 31% for depression, 42% for anxiety, 20% for obsessive-compulsive symptoms, and 40% for insomnia.^6,47

Is it true that fear, panic, and mental disorders occur? How significant is the mental, emotional disorder? Is it necessary to intervene? Who needs to be treated? These are questions that can be answered by monitoring and surveillance through measurement. Communities, families, and governments need the information to be able to act and care. Psychosocial surveillance can help identify health status, including mental illness due to the COVID-19 pandemic, and the need for the more effective and rational distribution of resources.⁵⁵ Several studies suggested that Cognitive Behavior Therapy (CBT) is provide good evidences can ameliorate this situation,^56,57 which also include internet CBT.⁵⁸

Psychosocial impact monitoring is conducted for surveillance and response purposes and for measuring the impact of disasters to decide a policy. The risks for face-to-face measurement are very high, apart from the restriction regulations themselves. Online surveys are the best choice since most people have access to internet⁵⁹ and can afford gadgets.

This study is limited to the results of measuring secondary data. A comparison with the results of direct measurements by experts will provide better evidence of the extent to which online surveys are reliable and trustworthy.⁶⁰ Other studies has informed that COVID-19 pandemic hemodynamic changes in the brain^61–64 which is beyond the context of this study.

Conclusion

The SRQ-20, PTSD, and CESD-10 questionnaires can be used as online instruments for assessing the psychological impact of the COVID-19 pandemic. The use of an online self-rating questionnaire to monitor the community’s psychosocial condition due to the pandemic’s impact should be conducted with caution. Researchers analyze not only the measurement results but also the instrument itself. Analysis with Rasch modeling is very instrumental and clearer to explain other facts behind the measurement results.

Disclosure

All authors have no financial interests or conflicts of interest in this study.