Validity and reliability testing of the Shatri Sinulingga psychosomatic test (SSPT) questionnaire as a screening instrument for psychosomatic disorders in Indonesia

Abstract

The incidence of psychosomatic disorders is increasing in Indonesia, and therefore screening instruments that are culturally appropriate for the Indonesian population are needed. The aim of this study was to assess the validity and reliability of the Shatri Sinulingga psychosomatic test (SSPT) questionnaire as a screening instrument for psychosomatic disorders in Indonesia. An analytic descriptive cross-sectional study divided into two stages (questionnaire formulation and distribution through the Psikosom.id application) was conducted in 2023. The validity test was carried out using the product moment technique, and Cronbach's alpha assessment was carried out to evaluate the reliability of the questionnaire. A total of 372 participants were included, with a mean age of 39.24 years old. The SSPT questionnaire scores had strong correlations with the scores of hospital anxiety and depression scale (HADS) (r = 0.668; p < 0.001) and the Symptom Checklist-90 (SCL-90) (r = 0.674; p < 0.001); and moderately correlated with the score of fatigue severity scale (FSS) (r = 0.505; p < 0.001) and Pittsburgh Sleep Quality Index (PSQI) (r = 0.492; p < 0.001). The correlation coefficient (r) value of each SSPT question ranged from 0.389–0.726, and all r higher than r table with an overall Cronbach's alpha of 0.744. The SSPT questionnaire's strong predictive performance: had a sensitivity of 75.6% and specificity of 71.4% with an area under the curve (AUC) of 81% (95%CI: 76.6–85.4%; p < 0.001) to HADS; sensitivity of 80% and specificity of 58.6% with an AUC of 75.1% (95%CI: 70–80.2%; p < 0.001) to FSS; sensitivity of 64% and specificity of 67.5% with an AUC of 71.2% (95%CI: 65.4–77%; p < 0.001) to PSQI; and had sensitivity of 78% and specificity of 67.4% with an AUC of 80.9% (95%CI: 76.6–85.3%; p < 0.001) to SCL-90. The study highlights that the SSPT questionnaire is valid and reliable to be used as a screening instrument for psychosomatic disorders in Indonesia.

Introduction

Psychosomatic disorders are physical conditions closely related to psychosocial factors that can influence both the onset and clinical course of the disease [1]. Studies showed that nearly 30% of patient visits to primary or secondary healthcare centers involve functional disorders or psychosomatic disorders, with symptoms primarily affecting the cardiovascular, digestive, musculoskeletal, and connective tissue systems [2–4]. Common complaints include non-specific pain, fatigue, tachycardia, and tinnitus [2]. Psychosomatic disorders not only reduce quality of life but also increase economic costs, while in patients with structural disorders, psychosomatic disorders can increase mortality rate [5,6].

The prevalence of psychosomatic disorders is estimated to be 5–7% of the general population, with higher female representation [7-10]. The prevalence increases to approximately 17% of the primary care patient population [11]. Data from the psychosomatic polyclinic at Dr. Cipto Mangunkusumo National Hospital, Jakarta, Indonesia, indicates a significant rise in outpatient visits over recent years [12]. In 2020, there were 388 visits, which spiked to 756 in 2021 and this trend continued with 818 visits in 2022 and 821 visits in 2023 [12]. Given this increasing burden, it has become essential to develop and implement effective screening instruments for psychosomatic disorders in the community.

Currently, several stress screening instruments have been translated into Indonesian [13–16]. However, many of these questionnaires, originally designed in other cultural contexts, may not be entirely suitable for the Indonesian population. Cultural nuances, especially in an Asian and predominantly Muslim society, often necessitate cross-cultural adaptation [17]. For instance, the Pittsburgh Sleep Quality Index (PSQI) includes a question about waking up in the middle of the night, which could lead to misleading results [18]. Many Indonesians participate in the Tahajud prayer, a common religious practice performed in the early hours of the morning, potentially leading to false-positive responses [19].

Moreover, there is currently no concise screening instrument available in Indonesia that comprehensively integrates physical, psychological, and social factors. To address this gap, we developed a new questionnaire, Shatri Sinulingga psychosomatic test (SSPT) questionnaire, specifically tailored to the Indonesian population to enable more accurate and culturally appropriate screening of psychosomatic disorders. The aim of this study was to assess the validity and reliability of the SSPT questionnaire in screening for psychosomatic disorders in Indonesia.

Methods

Study design, setting and sampling

A cross-sectional study was conducted to assess the validity and reliability of the SSPT questionnaire. Online surveys were distributed via the Psikosom.id application to Indonesian residents from February to May 2023. The minimum sample size required for the correlation analysis was 75 participants, calculated using a correlation formula [20], while the minimum sample size for the sensitivity and specificity analysis was 205 participants, determined using a sensitivity-specificity formula [21]. Participant recruitment was carried out using the consecutive sampling method.

Participants

This study included Indonesian residents aged 18 years or older who owned an Android-based smartphone with internet access. Eligible participants were also required to be willing to download the Psikosom.id application and provide informed consent before participating. Participants were excluded if they did not complete all the necessary data required for the study.

Questionnaire development

In December 2022, we initiated the development of a structured questionnaire to assess stress caused by specific stressors relevant to Indonesian habits and culture. This questionnaire was based on over 25 years of experience in the field of psychosomatic medicine and was complemented by a comprehensive literature review of various stress-induced illnesses and disorders [22]. Numerous screening instruments for psychosomatic disorders have been established; however, many of these instruments have challenges in terms of complexity and usability. For instance, the Symptom Checklist-90 (SCL-90) is lengthy, while others, like the Fatigue Severity Scale (FSS) cover too broad a range of scales, and the Pittsburgh Sleep Quality Index (PSQI) employs intricate scoring systems. Similarly, the hospital anxiety and depression scale (HADS) features a tiered approach that may not be user-friendly for the Indonesian population. Recognizing these limitations, we aimed to create a more accessible and concise questionnaire that would effectively screen for psychosomatic disorders among Indonesians.

Our structured questionnaire consists of closed-ended questions with simple ‘Yes’ or ‘No’ answer options. Each question carries equal scoring weight, with a ‘Yes’ response assigned a score of 1 and a ‘No’ response assigned a score of 0. This straightforward scoring system facilitates ease of use and interpretation. Following the formulation of the questions, discussions with experts were conducted to refine and finalize the questionnaire. To ensure its practicality, the SSPT questionnaire was tested on a sample of ten participants in January 2023. This pilot test aimed to identify any potential technical issues that could arise during the completion process. The feedback from this initial testing phase was important in making necessary adjustments to enhance the clarity and effectiveness of the questionnaire. The flowchart illustrating the development of the SSPT questionnaire is presented in Figure 1. The final version of the question items of the SSPT questionnaire is presented in Table 1.

Figure 1.

Flowchart of development processes of the Shatri Sinulingga psychosomatic test (SSPT) questionnaire.

Table 1.

The Shatri Sinulingga psychosomatic test (SSPT) questionnaire

No. No.	Question Pertanyaan	Answer Jawaban
		Yes Ya	No Tidak
1.	Do you have any chronic diseases? Apakah anda memiliki penyakit kronik?
2.	Do you often feel unwell? Apakah anda sering merasa kesehatan anda terganggu?
3.	Do you have any complaints that occur during emotional changes? Apakah anda memiliki keluhan yang terjadi saat perubahan emosi?
4.	Do you have complaints that keep alternating/moving, not settling? Apakah anda memiliki keluhan yang terus berganti-ganti/berpindah- pindah, tidak menetap?
5.	Do you often take medicine for aches and pains or headaches or traditional medicines? Apakah anda sering minum obat/jamu pegal linu atau sakit kepala?
6.	Do you often experience difficulties in your daily life? Apakah anda sering mengalami kesulitan dalam kehidupan sehari-har i?
7.	Do you often feel angry, upset, disappointed, frustrated, sad, etc.? Apakah anda sering merasa marah, kesal, kecewa, frustrasi, sedih, dll?
8.	Do you often have problems with family, friends, or neighbors? Apakah anda sering memiliki masalah dengan keluarga, teman, tetangga?
9.	Do you have trouble sleeping? Apak ah anda memi lik i k es ulit an/gangguan tid ur ?

Data collection

An online survey was conducted among Indonesian residents through the Psikosom.id application from February to May 2023. The collected data included sociodemographic information such as age, sex, province, occupation, education, ethnicity, religion, and marital status. Those who completed the SSPT questionnaire were required to complete the other questionnaires as well: SSPT, HADS, FSS, PSQI, and SCL-90. This was done to assess the correlation of the SSPT questionnaire with those instruments.

The HADS consists of 14 items divided into two subscales, including seven items that evaluate depression (HADS-D), while the other seven assess anxiety (HADS-A). Responses are rated on a 0–3 Likert scale, with higher scores indicating greater levels of anxiety and depression [23]. The FSS measures fatigue through nine questions rated on a Likert scale from 1–7, where 1 means “Strongly disagree,” 4 means “Neither agree nor disagree,” and 7 means “Strongly agree.” Higher scores on the FSS reflect increased fatigue [24]. The PSQI assesses sleep disorders and comprises nine questions grouped into seven domains, each with an unequal scoring system. A higher PSQI score indicates greater sleep disturbances [25]. Finally, the SCL-90 evaluates psychopathological symptoms through 90 questions scored on a 0–4 Likert scale. This instrument covers various domains, including somatization, obsessive-compulsive behavior, interpersonal sensitivity, depression, anxiety, anger-hostility, paranoid ideation, psychotic experiences, and additional items. Higher scores on the SCL-90 indicate a greater presence of psychopathological symptoms [26].

Statistical analysis

To assess the correlations between the SSPT questionnaire with SSPT, HADS, FSS, PSQI, and SCL-90, the Spearman correlation test was employed. The validity of the questions in the SSPT questionnaire measures was evaluated using corrected item-total correlation (Pearson's product moment correlation), and the reliability was determined through Cronbach's alpha, while the diagnostic accuracy test was measured using area under the curve (AUC) analysis. All statistical analyses were performed using SPSS software version 26 (SPSS Inc., Chicago, USA).

Results

Characteristics of the participants

A total of 615 individuals downloaded and registered for the Psikosom.id application (Figure 2). However, 14 participants were excluded from the study due to being under 18 years of age, leaving 601 participants. A total of 585 participants completed the SSPT questionnaire, 497 completed the HADS questionnaire, 426 completed the FSS questionnaire, 405 completed the PSQI questionnaire, and 392 completed the SCL-90 questionnaire. Among them, 372 participants completed all five questionnaires and were therefore included in the final analysis.

Figure 2.

Flowchart of the questionnaire completion.

A total of 372 participants were included in the study, as presented in Table 2. The mean age of the participants was 39 years, with the majority being women (63.2%). Of the participants, 74.7% were Muslim. Over 40% belonged to the Javanese or Batak ethnic groups. More than two- thirds of the participants were married, and over one-third were from North Sumatra, following DKI Jakarta (19.1%). Most participants had a higher education background, and only seven were not-employed (Table 2).

Table 2.

Characteristics of the participants (n = 372)

Characteristics	Frequency (%)
Age (years), mean±SD	39.24±10.18
Sex
Men	137 (36.8)
Women	235 (63.2)
Religion
Muslim	278 (74.7)
Christian	47 (12.6)
Catholic	17 (4.6)
Hindu	28 (7.5)
Buddha	2 (0.5)
Ethnicity
Aceh	11 (3.0)
Bali	17 (4.6)
Sunda	16 (4.3)
Batak	79 (21.2)
Betawi	2 (0.5)
Java	77 (20.7)
Melayu	20 (5.4)
Minang	23 (6.2)
Others	28 (7.5)
Not stated	99 (26.6)
Marital status
Single	101 (27.2)
Widow	13 (3.5)
Married	258 (69.4)
Province
Aceh	9 (2.4)
Bali	28 (7.5)
Bangka Belitung	2 (0.5)
Banten	8 (2.2)
Bengkulu	2 0.5)
Gorontalo	1 (0.3)
DKI Jakarta	71 (19.1)
Jambi	2 (0.5)
West Java	35 (9.4)
Central Java	15 (4)
East Java	11 (3)
North Sumatera	129 (34.7)
DI Yogyakarta	2 (0.5)
Riau	10 (2.7)
Riau Islands	2 (0.5)
Lampung	1 (0.3)
South Sumatera	3 (0.8)
South Sulawesi	2 (0.5)
Central Sulawesi	2 (0.5)
North Sulawesi	2 (0.5)
South Kalimantan	2 (0.5)
Central Kalimantan	1 (0.3)
Not stated	32 (8.6)
Education
Basic education (out of school, primary school, and junior high school)	0 (0.0)
Secondary education (senior high school)	46 (12.4)
High education (diploma to doctorate degree)	326 (87.6)
Employment status
Employed	365 (98.1)
Not-employed	7 (1.9)

Correlation between SSPT score with HADS, FSS, PSQI, and SCL-90 scores

The results of the correlation test indicated significant correlations between the SSPT score and the scores of all questionnaires (HADS, FSS, PSQI, and SCL-90), all had p < 0.001 (Table 3). The SSPT scores were strongly correlated with HADS and SCL-90 scores and moderately correlated with FSS and PSQI scores (Table 3).

Table 3.

Correlation between Shatri Sinulingga psychosomatic test (SSPT) questionnaire score with HADS, FSS, PSQI and SCL-90 (n = 372)

Questionnaire	Correlation coefficient (r)	p-value
Hospital anxiety and depression scale (HADS)	0.668	<0.001a*
Fatigue severity scale (FSS)	0.505	<0.001a*
Pittsburgh Sleep Quality Index (PSQI)	0.492	<0.001a*
Symptom Checklist-90 (SCL-90)	0.674	<0.001a*

^aAnalyzed using Spearman correlation

^*Statistically significant at p < 0.01

Validity and reliability test of the SSPT questionnaire

For this study, a question was considered valid if its correlation coefficient (r) exceeded the critical value from the r table, set at 0.1017 for a sample size of 372. Each question's r value was compared to this threshold, and those with r values greater than 0.1017 were deemed valid. Our data indicated that all nine SSPT questions met this criterion, confirming their validity (Table 4).

Table 4.

Validity test of each question of Shatri Sinulingga psychosomatic test (SSPT) questionnaire

No	Question	r valuea	r table	p-value
1	Do you have any chronic diseases? Apakah anda memiliki penyakit kronik?	0.389	0.1017	<0.001*
2	Do you often feel unwell? Apakah anda sering merasa kesehatan anda terganggu?	0.726	0.1017	<0.001*
3	Do you have any complaints that occur during emotional changes? Apakah anda memiliki keluhan yang terjadi saat perubahan emosi?	0.663	0.1017	<0.001*
4	Do you have complaints that keep alternating/moving, not settling? Apakah anda memiliki keluhan yang terus berganti- ganti/berpindah-pindah, tidak menetap?	0.563	0.1017	<0.001*
5	Do you often take medicine for aches and pains or headaches or traditional medicines? Apakah anda sering minum obat/jamu pegal linu atau sakit kepala?	0.486	0.1017	<0.001*
6	Do you often experience difficulties in your daily life? Apakah anda sering mengalami kesulitan dalam kehidupan sehari-hari?	0.572	0.1017	<0.001*
7	Do you often feel angry, upset, disappointed, frustrated, sad, etc.? Apakah anda sering merasa marah, kesal, kecewa, frustrasi, sedih, dll?	0.695	0.1017	<0.001*
8	Do you often have problems with family, friends, or neighbors? Apakah anda sering memiliki masalah dengan keluarga, teman, tetangga?	0.495	0.1017	<0.001*
9	Do you have trouble sleeping? Apakah anda memiliki kesulitan/gangguan tidur?	0.503	0.1017	<0.001*

^aAnalyzed using Pearson's product moment correlation

^*Statistically significant at p < 0.01

Following the validity assessment, the reliability of the questionnaire was evaluated using Cronbach's alpha. A Cronbach's alpha coefficient of 0.744 was obtained, which is well above the acceptable value of 0.5. This suggested that the SSPT questionnaire had good internal consistency and was reliable for measuring the intended constructs (Table 5).

Table 5.

Reliability test of each question of Shatri Sinulingga psychosomatic test (SSPT) questionnaire

Question	Scale means if the item deleted	Scale variance if the item deleted	Corrected item-total correlationa	Cronbach's alpha if the item deleted
1	4.79	20.010	0.296	0.739
2	4.63	18.286	0.666	0.707
3	4.55	18.621	0.562	0.715
4	4.84	19.181	0.587	0.721
5	4.83	19.588	0.448	0.730
6	4.77	19.062	0.542	0.721
7	4.62	18.382	0.637	0.709
8	4.86	19.587	0.490	0.728
9	4.75	19.347	0.450	0.727

^aAnalyzed using Cronbach's alpha

Diagnostic accuracy test of SSPT to predict anxiety-depression, fatigue, sleep disorders, and psychopathological symptoms

The SSPT score demonstrated strong predictive performance for anxiety-depression symptoms, as measured by the HADS. The area under the curve (AUC) was 0.81 (95%CI: 76.6−85.4%; p < 0.001), indicating high diagnostic accuracy (AUC >0.8). The optimal cut-off score of 2 was identified, with a sensitivity of 75.6% and a specificity of 71.4%, suggesting that the SSPT score was effective in screening for anxiety-depression symptoms (Figure 3A).

Figure 3.

Diagnostic accuracy test of SSPT to predict anxiety-depression, fatigue, sleep disorders, and psychopathological symptoms. (A) Receiver operating characteristic (ROC) curve of SSPT to predict anxiety and depression symptoms based on hospital anxiety and depression scale (HADS). (B) ROC curve of SSPT to predict fatigue based on fatigue severity scale (FSS). (C) ROC curve of SSPT to predict sleep disorders based on Pittsburgh Sleep Quality Index (PSQI). (D) ROC curve of SSPT to predict psychopathological symptoms based on Symptom Checklist-90 (SCL- 90).

For predicting fatigue, as assessed by the FSS, the SSPT score also performed well. The AUC was 0.75 (95%CI: 70−80.2%; p < 0.001), which suggested that the SSPT was a reasonably accurate instrument (AUC >0.7). The selected cut-off score of 2 provided a sensitivity of 80% and a specificity of 58.6%, suggesting moderate predictive ability for fatigue (Figure 3B).

The SSPT score's predictive capability for sleep disorders, evaluated using the PSQI, yielded an AUC of 0.71 (95%CI: 65.4−77%; p < 0.001), demonstrating acceptable accuracy (AUC >0.7) (Figure 3C). A cut-off score of 2 was determined, with a sensitivity of 64% and a specificity of 67.5%, suggesting that the SSPT score was reasonably effective in identifying individuals with sleep disorders.

Lastly, the SSPT score was found to be a strong predictor of psychopathological symptoms, as measured by the SCL-90, with an AUC of 0.81 (95%CI: 76.6−85.3%; p < 0.001), further supporting its high diagnostic value (AUC >0.8). A cut-off score of 2 provided a sensitivity of 78% and a specificity of 67.4%, suggesting that the SSPT score was a reliable screening instrument for identifying psychopathological symptoms (Figure 3D).

Discussion

This study involved young adult participants, with a higher proportion of females than males. Although most participants were from North Sumatra and DKI Jakarta, the samples also included individuals from various other provinces across Indonesia. This geographical diversity suggests that the sample was reasonably representative of the broader Indonesian population, particularly in terms of ethnic and cultural variation. However, the educational background of participants, primarily from higher education institutions, does not fully reflect the general population. According to the 2023 People's Welfare Statistics from the Central Statistics Agency, the majority of Indonesians have completed only primary education (59.62%), followed by secondary education (30.22%), with just 10.15% having pursued higher education [27]. This overrepresentation of participants with higher education levels may limit the generalizability of the findings to populations with lower educational attainment.

The validity and reliability of the SSPT questionnaire were key components of this study. Validity testing was conducted to determine whether the instrument accurately measured the intended constructs [28–30]. Reliability, encompassing internal consistency, stability, and equivalence [31], was also evaluated. Our findings confirmed that all nine items of the SSPT questionnaire were both valid and reliable, supporting its use for the screening of psychosomatic disorders.

Following this, the accuracy of the SSPT questionnaire in screening for psychosomatic symptoms was analyzed by comparing it with established screening instruments. The HADS is a widely used questionnaire for anxiety and depression screening, particularly in hospital settings [23]. In 2023, a study validated the HADS for the general Indonesian population, confirming its validity and reliability [32]. Furthermore, a study comparing HADS with the Zung self-rating anxiety/depression scale (SAS/SDS) in patients with non-small cell lung carcinoma, found that HADS was equally effective but quicker to administer [33]. In this present study, we observed a strong correlation between SSPT and HADS scores, reinforcing the validity of the SSPT in identifying anxiety and depression. With a cut-off point score of 2, the SSPT questionnaire demonstrated strong predictive power for screening these symptoms.

Similarly, the FSS was used to assess the SSPT's capacity to screen for fatigue. The FSS is a widely accepted instrument used in conditions such as systemic lupus erythematosus (SLE), osteoarthritis, and various chronic conditions, including cancer and neurological disorders [34]. The Indonesian version of the FSS has previously been validated in SLE patients, showing high reliability with a Cronbach's alpha 0.946 [35]. In our study, a moderate correlation between SSPT and FSS was observed, indicating that the SSPT could serve as a reasonable predictor of fatigue, with a cut-off point score of 2 for fatigue screening.

The PSQI was also employed to evaluate the SSPT's predictive accuracy for sleep disorders. The PSQI is a widely recognized instrument for assessing sleep quality over a one-month period [18]. A systematic review and meta-analysis confirmed the strong validity and reliability of the PSQI in both clinical and non-clinical populations [25]. In the Indonesian context, the PSQI has been validated in patients with chronic kidney disease, further supporting its reliability [36]. In our study, the SSPT showed a moderate correlation with PSQI scores, suggesting it is a fairly effective instrument for sleep disorder screening.

Additionally, we compared the SSPT with the SCL-90, a comprehensive instrument for assessing psychopathological symptoms [26]. The SCL-90 was translated into Indonesian and validated in 1994, showing good reliability and validity [37]. In our study, the SSPT showed a strong correlation with SCL-90 scores, further underscoring its utility in screening for anxiety and depression, with a cut-off point of 2.

This study benefits from a diverse participant pool from various regions across Indonesia, which enhances its reliability and relevance to the Indonesian population. However, there are some limitations. The use of an Android-based application may have excluded individuals without access to smartphones, potentially introducing selection bias. Moreover, the study did not perform a test-retest reliability assessment, which is important for measuring the stability of the instrument over time. Future studies should address these limitations to further validate the SSPT's applicability across a broader spectrum of the population.

Conclusion

In this study, we demonstrated that the SSPT questionnaire is a valid and reliable instrument for screening psychosomatic disorders in Indonesia. The integration of the SSPT questionnaire into routine clinical practice could enhance overall psychosomatic care.

Ethics approval

This study received ethical clearance from Ethical Committee Faculty of Medicine, Universitas Indonesia – Cipto Mangunkusumo National General Hospital, Jakarta, Indonesia, with number: KET-9/UN2.F1/ETIK/PPM.00.02/2023.

Competing interests

The authors declare that there is no conflict of interest.

Funding

This study was supported by grant from the Ministry of Education, Culture, Research and Technology of the Republic of Indonesia Grant 2022 (SKU-137/UN2.R3/HKP.05/2022), Faculty of Medicine Universitas Indonesia.

Underlying data

Derived data supporting the findings of this study are available from the corresponding author on request.

How to cite

Shatri H, Sinulingga DI, Faisal E, et al. Validity and reliability test of Shatri Sinulingga psychosomatic test (SSPT) questionnaire as a screening instrument for psychosomatic disorders in Indonesia. Narra J 2024; 4 (3): e1373 - http://doi.org/10.52225/narra.v4i3.1373.