ABSTRACT
Objectives
This study aimed to compared Patient‐Reported Outcomes Measurement Information System (PROMIS) anxiety, depression, and anger item bank among Korean, US and Dutch general population.
Methods
Between December 2021 and January 2022, we surveyed representative Korean participants (N = 2699). Then we compared the mean T‐scores of PROMIS anxiety, depression, and anger full items bank among Korean, US (N = 1696) and the Dutch (N = 1002) populations. Differential item‐functioning (DIF) analyses were also performed. We also compared each score by age group, sex, presence of comorbidities, and general health status.
Results
In Korean, the mean T‐scores for anxiety, depression, and anger were 45.3 (standard deviation [SD] = 11.6), 48.4 (SD = 11.2), and 44.9 (SD = 12.6), respectively. Among the general population in Korea, patients aged 35–44 years and those with comorbidities had higher anxiety, depression, and anger scores. In the DIF analyses between the US and Korean populations, 28%, 32%, and 45% were flagged for uniform or non‐uniform DIF in anxiety, depression and anger, respectively.
Conclusions
Considering the cultural differences, we recommend using a harmonized approach that includes country‐specific reference values while retaining a standardized core set of items to enable cross‐country comparability.
Keywords: emotional distress, general population, patient‐reported outcomes, PROMIS
1. Introduction
Given the high prevalence of anxiety and depression in patients with serious disease such as cancer (Blatch Armon et al. 2023; Dong et al. 2024), clinicians tried to routinely measure and manage these conditions (Terwee et al. 2021). However, accumulating evidence suggests that minor stressors in daily life also could lead to psychological disorders (Johnsson, Zolkowska, and McNeil 2015; Monzonís‐Carda et al. 2021; Casey et al. 2022). Thus, negative emotional problems may also occur in the general population, even in the absence of specific events or disease.
Patient‐reported outcome measures (PROMs) are the most appropriate methods to measure emotional distress (Shadmi et al. 2018). Despite the use of numerous PROMs in practice (Thornicroft et al. 2014; Gelkopf, Mazor, and Roe 2021) many of them measure the same or a similar construct (Schalet et al. 2016) and lack reliability and validity (Gelkopf, Mazor, and Roe 2021). To improve existing measures, the Patient‐Reported Outcomes Measurement Information System (PROMIS) was developed, which covers a broad range of relevant domains and provides strong evidence for its validity and reliability in a broad range of populations (Terwee et al. 2021). Furthermore, PROMIS emotional distress provides support for transforming raw scores to a mean of 50 and a standard deviation (SD) of 10 using the US general population as a reference (Pilkonis et al. 2011), and the reference values help support the use of PROMIS in daily clinical practice and research.
However, some studies have shown that the reference values of the PROMIS scales in other countries deviate from the mean score of 50 obtained from the US general population (Elsman et al. 2021). The PROMIS guidelines recommend differential item functioning (DIF) analyses to test whether people from different groups with the same construct respond differently to an item (Embretson et al. 2013; Teresi et al. 2007). Although several studies have calculated DIF in European (EU) populations, few have compared Asian populations, including Korea (E. Elsman et al. 2022). Thus, we compared the PROMIS anxiety, depression, and anger item banks among Korean, US, and Dutch general population and obtained a Korean‐specific reference value by the socio‐demographic differences.
2. Methods
2.1. Participants
We conducted a national survey of 2699 participants aged 19–84 years using proportional stratified sampling across 17 provinces in the Republic of Korea between December 2021 and January 2022. The study was approved by the Institutional Review Board of Samsung Medical Center (IRB number: 2021‐03‐005). Written informed consent was obtained from all the participants.
In this study, we used US and Dutch databases. For the representative US population, we used PROMIS Wave 1 data obtained from the Health Measures Dataverse repository (Cella et al. 2010). PROMIS Wave 1 data were collected from the general US population and populations with multiple diseases from July 2006 to March 2007. Specifically, we used data from Forms A (n = 887) and B (n = 889) (Cella et al. 2010), which included the emotional distress. The total number of participants was 1, 696. The Dutch’ data were obtained from a previous study (E. Elsman et al. 2022). The data were collected in 2014. Participants were recruited from an existing internet panel of the Dutch general population using a data collection company (Desan Research Solutions) (E. Elsman et al. 2022). Representativeness of the participants was compared with data from Statistics Netherlands in 2013, with a maximum allowable deviation of 2.5% (Flens et al. 2017). The total number of Dutch was 1002.
2.2. Measurement
Although the online survey was the primary instrument used, a face‐to‐face survey also was conducted with individuals aged 65 and older due to concerns regarding digital health literacy. This was done to avoid missing a potentially significant group. A previous study using PROMIS did not find mode effects between the web, mail, and telephone surveys. The survey comprised the PROMIS Anxiety item bank v1.0, Depression item bank v1.0, and Anger item bank v1.1. The average response time was 30 min.
The PROMIS Anxiety Item bank v1.0 included four short forms of different lengths: SF4a, SF6a, SF7a, and SF8a, which had four, six, seven, and eight items, respectively. The PROMIS Depression item bank v1.0, included four short forms: SF4a, SF6a, SF8a, and SF8b, with four, six, eight, and eight items, respectively. The PROMIS Anger item bank v1.1 included one short form, SF5a, with five items. Given the widespread use of short‐form in numerous settings, a comparative analysis was performed to evaluate the comparability between the full bank and the short form.
All items were five‐point Likert response scale (1 = never, 2 = rarely, 3 = sometimes, 4 = often, and 5 = always). The recall period for all items was “In the past 7 days.” We converted the raw scores into T‐scores standardized for the general US population (mean (SD) 50 (10)) using the conversion table provided by the Assessment Center (https://www.assessmentcenter.net; Northwestern University, Evanston, IL, USA) (Cella et al. 2010). Lower scores indicated lower levels of emotional distress. Each short‐form score was calculated individually. We collected data on age, sex, region of residence, marital status, educational level, employment status, and comorbidities.
2.3. Statistical Analyses
Descriptive statistics were used to summarize the participants' sociodemographic characteristics and responses to the PROMIS emotional distress items. We compared the mean T‐scores of the Dutch and US populations and their subgroups.
DIF analyses were used to evaluate whether people from different populations with similar levels of emotional distress responded similarly to these items. The absence of DIF allowed valid comparisons of T‐scores between populations. The DIF was evaluated using the R package Lordif (version 0.3–3), which employs an ordinal logistic regression framework. There are two types of DIF: uniform and non‐uniform. A uniform DIF exists if its magnitude is consistent across the entire range of the assessed traits. Nonuniform DIF exists if the magnitude of the DIF varies across different trait levels. Because statistical power is dependent on sample size, the difference in population parameters was significant given the sufficiently large sample size. In response to this concern (S. W. Choi, Gibbons, and Crane et al. 2011), we used McFadden's pseudo R 2‐change stricter threshold 0.02 as the critical value to flag for possible DIF (S. W. Choi, Gibbons, and Crane et al. 2011). This method of analysis involved three models to assess the DIF.
Then, the T‐scores of the anxiety, depression, and anger, and each short forms compared by age group (18–34, 35–44, 45–54, 55–64, 65–74, and ≥ 75 years), gender, comorbidities, and general health within the Korean population using the t‐test. Multivariate linear regression was performed to assess the factors associated with emotional distress. Age, sex, marital status, educational level, income level, region of residence, current employment status, number of chronic diseases, and disabilities were adjusted for as covariates.
Statistical significance was set at p < 0.05, and two‐tailed tests were used for all calculations. Statistical analyses were performed using the R 4.1.2 (R Foundation for Statistical Computing, Vienna, Austria).
3. Results
The mean age was 48.4 years (SD = 16.6), with a range of 18–85 years. Of the participants, 87.29% had more than a high school education and 65.51% reported having at least one chronic condition (Table 1).
TABLE 1.
Sociodemographic characteristics of study participants.
| Variables | Korean population (n = 2699) | US participants (n = 1696) | Dutch population (n = 1002) |
|---|---|---|---|
| Age, years | 48.4 (16.6) | 51.0 (18.6) | |
| 18–34 | 665 (24.64) | 402 (23.7) | 253 (25.25) |
| 35–44 | 487 (18.04) | 268 (15.8) | 147 (14.67) |
| 45–54 | 548 (20.3) | 282 (16.63) | 173 (17.27) |
| 55–64 | 507 (18.78) | 276 (16.27) | 216 (21.56) |
| 65–74 | 313 (11.6) | 197 (11.62) | 191 (19.06) |
| ≥ 75 | 179 (6.63) | 271 (15.98) | 22 (2.20) |
| Gender | |||
| Male | 1357 (50.28) | 833 (49.12) | 480 (47.90) |
| Female | 1342 (49.72) | 863 (50.88) | 522 (52.10) |
| Education level | |||
| ≤ Middle school | 343 (12.71) | 39 (2.3) | 321 (32.04) |
| High school | 461 (17.08) | 299 (17.63) | 400 (39.92) |
| ≥ Colleges | 1895 (70.21) | 1355 (79.89) | 281 (28.04) |
| Unknown | 0 (0) | 3 (0.18) | |
| Marital status | |||
| Single | 870 (32.23) | 278 (16.39) | |
| Married or living together | 1617 (59.91) | 1099 (64.8) | |
| Divorced | 80 (2.96) | 206 (12.15) | |
| Widowed | 132 (4.89) | 112 (6.6) | |
| Income level (a year) | |||
| Less than $20,000 | 515 (19.08) | 177 (10.44) | |
| $20,000–$49,999 | 1070 (39.64) | 585 (34.49) | |
| $50,000–$99,999 | 914 (33.86) | 659 (38.86) | |
| More than $100,000 | 200 (7.41) | 222 (13.09) | |
| Unknown | 0 (0) | 53 (3.13) | |
| Current employment status, employed | 1801 (66.73) | 791 (46.64) | |
| Chronic disease | |||
| Cardiovascular disease | 809 (29.97) | 647 (38.15) | |
| Endocrine diseases | 357 (13.23) | 172 (10.14) | |
| Cerebrovascular disease | 57 (2.11) | 61 (3.6) | |
| Musculoskeletal disease | 520 (19.27) | 364 (21.46) | |
| Respiratory disease | 422 (15.64) | 261 (15.39) | |
| Gastrointestinal disease | 269 (9.97) | 73 (4.3) | |
| Cancer | 87 (3.22) | 126 (7.43) | |
| Psychiatric | 222 (8.23) | 526 (31.01) | |
| No | 931 (34.49) | 395 (23.29) | |
Note: Values were presented n (%) or mean (SD).
The anxiety T‐score in the Korean population was 45.3 (SD = 11.6) with a range of 31.6–88.4. The anxiety T‐score of the Korean population was lower than the mean score of 48.5 in the US and 49.9 in the Dutch (Table 2). In most age and sex groups, excluding those with comorbidities, anxiety T‐scores in nearly all the short forms were < 50 (Supporting Information S1: Table S2). Regarding the distribution of anxiety T‐scores, 21% of the Korean population reported a score of 31.6, with the average score being lower than that of the US population because of the higher proportion of individuals with the lowest scores (Figure 1A).
TABLE 2.
Emotional distress score by nation and age, gender and comorbidities.
| Korean population (n = 2699) | US participants (n = 1696) | Korean versus US | Dutch population (n = 1002) | Korean versus Dutch | |
|---|---|---|---|---|---|
| Categories | Mean (SD) | Mean (SD) | p‐value | Mean (SD) | p‐value |
| Anxiety | |||||
| Overall | 45.3 (11.6) | 48.5 (9.8) | < 0.001 | 49.9 (10.1) | < 0.001 |
| Age | |||||
| 18–34 | 46.7 (12.9) | 51.9 (10.1) | < 0.001 | 51.8 (9.9) | < 0.001 |
| 35–44 | 47.1 (12.5) | 49.6 (10.7) | 0.032 | 51.4 (10.9) | < 0.001 |
| 45–54 | 45.7 (11.5) | 49.6 (9.6) | < 0.001 | 50.0 (10.9) | < 0.001 |
| 55–64 | 44.8 (10.6) | 47.6 (8.8) | 0.002 | 48.9 (9.4) | < 0.001 |
| 65–74 | 40.7 (8.4) | 44.2 (8.9) | 0.001 | 47.5 (9.1) | < 0.001 |
| ≥ 75 | 43.3 (9.5) | 45.2 (8.2) | 0.059 | 46.2 (9.4) | < 0.001 |
| Gender | |||||
| Male | 44.8 (11.8) | 47.4 (9.8) | < 0.001 | 49.0 (10.0) | < 0.001 |
| Female | 45.7 (11.4) | 49.5 (9.8) | < 0.001 | 50.6 (10.1) | < 0.001 |
| Comorbidity | |||||
| No | 42.5 (10.9) | 46.5 (8.8) | < 0.001 | ||
| Cardiovascular disease | 46.1 (11.4) | 47.7 (9.7) | 0.016 | ||
| Endocrine diseases | 46.7 (11.9) | 49.8 (9.5) | 0.012 | ||
| Cerebrovascular disease | 53.3 (13.7) | 47.0 (7.7) | 0.010 | ||
| Musculoskeletal disease | 48.4 (11.8) | 48.6 (9.3) | 0.820 | ||
| Respiratory disease | 48.8 (12.6) | 49.5 (10.6) | 0.538 | ||
| Gastrointestinal disease | 50.2 (13.0) | 49.4 (9.9) | 0.637 | ||
| Cancer | 46.0 (12.6) | 47.8 (9.0) | 0.291 | ||
| Psychiatric | 57.9 (12.9) | 53.3 (9.9) | < 0.001 | ||
| Depression | |||||
| Overall | 48.4 (11.2) | 49.4 (9.7) | 0.016 | 49.6 (10.0) | < 0.001 |
| Age | |||||
| 18–34 | 49.8 (12.7) | 52.8 (10.2) | 0.001 | 52.0 (9.3) | < 0.001 |
| 35–44 | 49.3 (11.8) | 49.5 (10.0) | 0.882 | 50.5 (10.8) | < 0.001 |
| 45–54 | 48.4 (11.0) | 50.4 (9.6) | 0.038 | 50.0 (11.0) | < 0.001 |
| 55–64 | 48.1 (10.4) | 48.8 (9.1) | 0.439 | 48.8 (9.8) | 0.001 |
| 65–74 | 44.8 (8.8) | 45.7 (8.9) | 0.416 | 47.0 (8.9) | < 0.001 |
| ≥ 75 | 47.5 (9.0) | 46.3 (7.6) | 0.213 | 46.0 (9.6) | < 0.001 |
| Gender | |||||
| Male | 47.8 (11.4) | 48.5 (9.5) | 0.199 | 48.8 (10.1) | < 0.001 |
| Female | 48.9 (11.1) | 50.3 (9.8) | 0.021 | 50.4 (9.9) | < 0.001 |
| Comorbidity | |||||
| No | 45.5 (10.8) | 46.9 (8.0) | 0.049 | ||
| Cardiovascular disease | 49.3 (10.6) | 48.5 (9.7) | 0.292 | ||
| Endocrine diseases | 50.1 (10.9) | 50.6 (10.1) | 0.697 | ||
| Cerebrovascular disease | 54.3 (11.9) | 48.6 (9.4) | 0.023 | ||
| Musculoskeletal disease | 51.5 (11.0) | 50.5 (9.4) | 0.263 | ||
| Respiratory disease | 51.8 (12.1) | 50.3 (10.6) | 0.192 | ||
| Gastrointestinal disease | 52.1 (12.5) | 50.6 (9.4) | 0.370 | ||
| Cancer | 48.6 (11.7) | 48.4 (9.9) | 0.943 | ||
| Psychiatric | 59.7 (11.6) | 54.3 (10.0) | < 0.001 | ||
| Anger | |||||
| Overall | 44.9 (12.6) | 47.4 (9.1) | < 0.001 | ||
| Age | |||||
| 18–34 | 47.4 (13.6) | 49.8 (9.7) | 0.006 | ||
| 35–44 | 48.0 (13.0) | 50.3 (8.7) | 0.014 | ||
| 45–54 | 45.7 (12.0) | 47.1 (8.6) | 0.087 | ||
| 55–64 | 43.9 (11.8) | 46.5 (9.1) | 0.005 | ||
| 65–74 | 38.0 (8.9) | 44.2 (8.5) | < 0.001 | ||
| ≥ 75 | 40.2 (11.0) | 44.5 (8.1) | < 0.001 | ||
| Gender | |||||
| Male | 44.5 (12.7) | 47.2 (9.3) | < 0.001 | ||
| Female | 45.4 (12.5) | 47.6 (9.0) | < 0.001 | ||
| Comorbidity | |||||
| No | 42.4 (11.9) | 46.1 (9.1) | < 0.001 | ||
| Cardiovascular disease | 44.6 (12.7) | 47.1 (9.0) | < 0.001 | ||
| Endocrine diseases | 45.5 (13.0) | 47.4 (10.0) | 0.124 | ||
| Cerebrovascular disease | 52.7 (14.8) | 44.9 (7.5) | 0.001 | ||
| Musculoskeletal disease | 47.9 (13.3) | 47.5 (9.2) | 0.646 | ||
| Respiratory disease | 49.4 (13.3) | 48.2 (9.3) | 0.270 | ||
| Gastrointestinal disease | 50.4 (13.5) | 46.7 (9.9) | 0.053 | ||
| Cancer | 45.7 (13.6) | 44.0 (7.6) | 0.354 | ||
| Psychiatric | 57.3 (13.8) | 51.2 (8.9) | < 0.001 | ||
FIGURE 1.
Distributions of T‐scores (based on full item bank) in the US and Korean populations. Panels (A–C) illustrate the distributions of Anxiety, Depression, and Anger T‐scores in the US and Korean populations, respectively.
Depression and anger also showed similar trends to anxiety. Depression T‐score in the Korean population was 48.4 (SD = 11.2; range = 33.5–86.4). The depression T‐score of the Korean population was also lower than that of the US and the Dutch (Table 2). Regarding the distribution of depression T‐scores, 20% of the Korean population reported a score of 33.5 (Figure 1B). Anger T‐score in the Korean population was 44.9 (SD = 12.6; range = 28.5–88.9), and it was also lower scores than US population (Table 2). Regarding the distribution of anger T‐scores, 18% of the Korean population reported a score of 28.5 (Figure 1C).
Among the Korean population, younger individuals, particularly those aged 35–44, showed significantly higher levels of emotional distress than older age groups, and the presence of comorbidities was also found to be associated with elevated anxiety, depression, and anger. Specifically, individuals with lower incomes and widows were significantly more anxious than those who were married or living together. Lower income and widowhood increased depression, whereas anger was more prevalent in females and employed individuals (Table 3, Figure 2).
TABLE 3.
Factors associated with emotional distress among the general population.
| Variables | Anxiety | Depression | Anger | |||
|---|---|---|---|---|---|---|
| Crude coefficient (95% CI) | Adjusted a coefficient (95% CI) | Crude coefficient (95% CI) | Adjusted a coefficient (95% CI) | Crude coefficient (95% CI) | Adjusted a coefficient (95% CI) | |
| Age, years | ||||||
| 18–34 | 3.40 (1.51, 5.29) | 9.87 (6.77, 12.97) | 2.27 (0.43, 4.11) | 7.52 (4.53, 10.51) | 7.20 (5.19, 9.22) | 13.02 (9.71, 16.33) |
| 35–44 | 3.80 (1.84, 5.76) | 10.29 (7.32, 13.25) | 1.82 (−0.09, 3.73) | 8.17 (5.31, 11.03) | 7.79 (5.70, 9.88) | 13.11 (9.94, 16.27) |
| 45–54 | 2.37 (0.43, 4.30) | 8.69 (5.80, 11.59) | 0.87 (−1.01, 2.75) | 7.27 (4.48, 10.07) | 5.47 (3.42, 7.53) | 10.61 (7.52, 13.71) |
| 55–64 | 1.52 (−0.43, 3.48) | 7.16 (4.32, 10.00) | 0.59 (−1.31, 2.49) | 6.30 (3.56, 9.04) | 3.68 (1.60, 5.76) | 8.16 (5.13, 11.19) |
| 65–74 | −2.62 (−4.72, −0.51) | 0.18 (−2.15, 2.50) | −2.72 (−4.76, −0.67) | 0.23 (−2.01, 2.47) | −2.24 (−4.48, 0.00) | −0.05 (−2.53, 2.43) |
| ≥ 75 | Reference | Reference | Reference | Reference | Reference | Reference |
| Gender | ||||||
| Male | Reference | Reference | Reference | Reference | Reference | Reference |
| Female | 0.92 (0.04, 1.79) | 0.99 (0.11, 1.87) | 1.17 (0.32, 2.02) | 1.01 (0.17, 1.86) | 0.89 (−0.06, 1.84) | 1.28 (0.35, 2.22) |
| Education level | ||||||
| ≤ Middle school | −2.53 (−3.86, −1.20) | 3.29 (0.99, 5.59) | −1.47 (−2.76, −0.17) | 2.48 (0.26, 4.69) | −5.74 (−7.17, −4.30) | 2.91 (0.46, 5.36) |
| High school | −0.96 (−2.13, 0.22) | 1.56 (0.24, 2.87) | −0.01 (−1.15, 1.14) | 1.62 (0.35, 2.89) | −2.12 (−3.39, −0.86) | 1.62 (0.22, 3.03) |
| ≥ Colleges | Reference | Reference | Reference | Reference | Reference | Reference |
| Marital status | ||||||
| Single | 2.44 (1.48, 3.39) | 0.68 (−0.63, 1.99) | 3.15 (2.24, 4.07) | 2.12 (0.85, 3.38) | 3.36 (2.33, 4.40) | 0.40 (−1.00, 1.80) |
| Married or living together | Reference | Reference | Reference | Reference | Reference | Reference |
| Divorced | 2.43 (−0.16, 5.03) | −0.02 (−2.56, 2.53) | 3.88 (1.38, 6.38) | 1.30 (−1.16, 3.75) | 1.70 (−1.11, 4.51) | −0.87 (−3.59, 1.84) |
| Widowed | 1.11 (−0.94, 3.16) | 2.76 (0.41, 5.10) | 2.96 (0.98, 4.93) | 3.62 (1.36, 5.89) | −1.34 (−3.56, 0.88) | 2.50 (0.00, 5.00) |
| Income level (a year) | ||||||
| Less than $20,000 | 0.95 (−0.95, 2.85) | 2.63 (0.63, 4.62) | 3.10 (1.27, 4.93) | 3.80 (1.88, 5.72) | −1.01 (−3.07, 1.04) | 1.97 (−0.15, 4.10) |
| $20,000–$49,999 | 1.03 (−0.72, 2.78) | 2.02 (0.28, 3.75) | 2.86 (1.16, 4.55) | 3.46 (1.79, 5.14) | 0.49 (−1.41, 2.39) | 1.93 (0.08, 3.78) |
| $50,000–$99,999 | 1.11 (−0.67, 2.89) | 1.16 (−0.55, 2.87) | 1.80 (0.09, 3.52) | 1.87 (0.23, 3.52) | 1.64 (−0.29, 3.56) | 1.71 (−0.11, 3.53) |
| $100,000 or more | Reference | Reference | Reference | Reference | Reference | Reference |
| Current employment status | ||||||
| Unemployed | Reference | Reference | Reference | Reference | Reference | Reference |
| Employed | 0.32 (−0.61, 1.25) | 0.66 (−0.32, 1.64) | −0.83 (−1.73, 0.07) | −0.28 (−1.22, 0.66) | 1.42 (0.41, 2.43) | 1.42 (0.38, 2.47) |
| Chronic disease | ||||||
| Cardiovascular disease | 1.15 (0.19, 2.10) | 3.47 (2.43, 4.52) | 1.28 (0.36, 2.21) | 3.28 (2.28, 4.29) | −0.41 (−1.45, 0.63) | 2.96 (1.85, 4.08) |
| Endocrine diseases | 1.60 (0.31, 2.89) | 2.78 (1.49, 4.08) | 2.03 (0.78, 3.28) | 2.97 (1.72, 4.23) | 0.66 (−0.74, 2.06) | 2.69 (1.31, 4.08) |
| Cerebrovascular disease | 8.14 (5.11, 11.17) | 8.84 (5.84, 11.83) | 6.09 (3.15, 9.03) | 6.61 (3.71, 9.51) | 7.95 (4.65, 11.24) | 9.21 (6.02, 12.40) |
| Musculoskeletal disease | 3.86 (2.75, 4.96) | 4.53 (3.42, 5.64) | 3.90 (2.84, 4.97) | 4.37 (3.30, 5.44) | 3.74 (2.54, 4.94) | 5.04 (3.86, 6.22) |
| Respiratory disease | 4.20 (3.01, 5.40) | 3.45 (2.24, 4.66) | 4.05 (2.89, 5.21) | 3.54 (2.37, 4.71) | 5.28 (3.98, 6.58) | 3.83 (2.55, 5.12) |
| Gastrointestinal disease | 5.49 (4.04, 6.94) | 5.45 (4.00, 6.89) | 4.15 (2.74, 5.55) | 4.43 (3.03, 5.83) | 6.12 (4.55, 7.69) | 5.68 (4.13, 7.22) |
| Cancer | 0.74 (−1.74, 3.22) | 1.07 (−1.39, 3.54) | 0.20 (−2.20, 2.60) | 0.21 (−2.17, 2.60) | 0.80 (−1.90, 3.49) | 1.77 (−0.86, 4.40) |
| Psychiatric | 13.75 (12.25, 15.26) | 13.04 (11.52, 14.56) | 12.38 (10.91, 13.85) | 11.64 (10.16, 13.12) | 13.44 (11.79, 15.10) | 12.22 (10.58, 13.86) |
Adjusted for age, sex, marital status, education level, income level (years), current employment status, and presence of chronic diseases.
FIGURE 2.
Proportions of normal, mild, moderate, and severe emotional distress, as defined by the Health Measures Dataverse, by patient characteristics. Panels (A–C) illustrate the proportions of the Korean population across T‐score cutpoints for Anxiety, Depression, and Anger, respectively.
In the DIF analyses between the US and Korean populations, 8 of 29 items (28%), 9 of 28 items (32%), and 10 of 22 items (45%) were flagged for uniform or non‐uniform DIF in anxiety, depression and anger, respectively. The R 2‐change values for each item are provided in Supporting Information S1: Table S1. Specifically, EDANX26 (“I felt fidgety”), EDANX37 (“I worried about other people's reactions to me”), EDANX44 (“I had twitching or trembling muscles”), EDANX46 (“I felt nervous”), EDANX49 (“I had difficulty in sleeping”), EDANX51 (“I had trouble relaxing”), EDANX54 (“I felt tense”) flagged for DIF in anxiety item bank. In the depression item bank, EDDEP06 (“I felt helpless”), EDDEP17 (“I felt sad”), EDDEP36 (“I felt unhappy”), EDDEP39 (“I felt I had no reason for living”), EDDEP46 (“I felt pessimistic”) were flagged for DIF (Figure 3).
FIGURE 3.
Graphical display of the items that exhibited differential item functioning between the US and Korean populations.
4. Discussion
This multinational study, based on a representative population, revealed that the Korean sample had lower emotional distress than the US and Dutch samples. Among Koreans, younger age, comorbidities, lower income, and widowhood were associated with significantly higher emotional distress than others across. Several items across the anxiety, depression, and anger scales demonstrated uniform or non‐uniform DIF, indicating potential cultural differences in the expression or perception of emotional distress.
Although there were similar scores between the US and Dutch groups, the Korean samples had lower mean T‐scores for anxiety, and anger than the US and Dutch samples. Previous studies have consistently demonstrated comparable levels of emotional distress between US and Dutch populations (E. Elsman et al. 2022). The lower scores for anxiety, depression, and anger among Koreans were similar to those found in previous studies. Even within the US, Asian Americans consistently report the symptoms of anxiety disorders less frequently than any other racial group (Hofmann et al. 2014). However, considering German study found lower anxiety in the German population relative to the US population (Abdel‐Khalek et al. 2009), this reflects the complex interplay between cultural, psychological, and methodological factors. First, the cultural norms in Korea, which often involve significant stigma around mental health issues and a collective priority over individual expression, might inhibit the overt acknowledgment and expression of emotional distress (Zhang et al. 2019). Second, differences in healthcare access and mental health service availability between these countries could have led to reporting discrepancies (Andrade et al. 2014). To address these issues, further qualitative research is necessary to understand the contextual meaning of how different groups respond to these items. In addition, the generation of core items based on items that have demonstrated low DIF may be necessary to assess emotional states in culturally diverse populations, as it can improve the validity of the instrument in international contexts.
Consistent with previous research, this study found that age, socioeconomic status, and comorbidities significantly affected emotional functioning. The higher levels of emotional distress reported among younger individuals, particularly those aged 35–44, may reflect the unique social, economic, and familial pressures facing this demographic, such as career establishment, financial responsibilities, and parenting (Gondek et al. 2024). The findings indicate the need for targeted mental health support services in this age group. Furthermore, the universal elevation of distress scores across anxiety, depression, and anger in individuals with comorbidities underscores the intertwined nature of their physical and mental health (Blatch Armon et al. 2023; Dong et al. 2024). This relationship highlights the necessity for integrated care approaches that address both physical and psychological aspects of health, especially in clinical settings where patients are being treated for chronic physical conditions (Rodgers et al. 2018).
In fact, we found that 28%, 32%, and 45% of the anxiety, depression, and anger items, respectively, were flagged for uniform or non‐uniform DIF. The flagged items suggest that respondents from different populations may not interpret the severity of their symptoms consistently. A previous study indicated that the construct “anger suppression” may be defined as “keeping anger inside but not letting it out” in the US, whereas in Korea it may be defined as “avoiding eye contact or staying silent to avoid getting angry.” (Chon et al. 2023a). Similarly, expressing anger may be defined as “slamming a door, throwing things or yelling” in the United States, whereas in Korea it may be defined as “blaming the other person.” (Chon et al. 2023b). Even with accurate translations, the connotations of words and phrases can vary between languages. Although the PROMIS emotional distress item banks were linguistically validated, the validation was performed in a pediatric setting rather than in an adult population (H. Choi et al. 2019). Given that this item may not fully capture how emotions are expressed in Korean culture, it is essential to conduct a more comprehensive cross‐cultural validity study on emotional distress in Asian populations.
This study had several limitations. First, our comparison with data from the US and Netherlands was indirect. In particular, the US data were collected 20 years ago; therefore, these differences cannot be overlooked. Further studies are necessary to collect data concurrently, using consistent methods and protocols, to ensure more accurate comparisons. Second, the data were collected in 2021 during the COVID‐19 pandemic. Consequently, the current population's level of emotional distress and their participation rates may differ. Ideally, reference values should be periodically updated to accurately reflect the latest population status.
4.1. Conclusion
In this study, we found that there were some differences in item function across cultures, particularly in the Asian context. The detection of DIF in 40% of the items indicates potential variability in item interpretation across cultures. We then calculated the average T‐scores segmented by age, sex, and the presence of comorbidities within the Korean general population. Given that several DIF were observed between the US and Korean populations, our data are valuable references for researchers to compare their study groups with those of the general Korean population. To maintain comparability across countries, the use of a standardized core set of items that did not show significant DIF could be considered.
Author Contributions
Jiseon Lee: conceptualization, data curation, formal analysis, investigation, writing–original draft. Yeonjung Lim: formal analysis, investigation. Dong Gi Seo: writing–review & editing. Minji K. Lee: writing–review & editing. Benjamin D. Schalet: writing–review & editing. Felix Fischer: writing–review & editing. Matthias Rose: writing–review & editing. Danbee Kang: conceptualization, data curation, formal analysis, funding acquistion, writing–original draft. Juhee Cho: conceptualization, project administration, supervision, writing–review and editing.
Ethics Statement
The study was approved by the Institutional Review Board of Samsung Medical Center (IRB number: 2021‐03‐005).
Consent
Informed consent was obtained from all individual participants included in the study.
Conflicts of Interest
The authors declare no conflicts of interest.
Supporting information
Supporting Information S1
Acknowledgments
The authors have nothing to report.
Funding: This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2020R1I1A2074210).
Jiseon Lee and Yeonjung Lim equally contributed to this work as first authors.
Contributor Information
Danbee Kang, Email: dbee.kang@skku.edu.
Juhee Cho, Email: jcho@skku.edu.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
- Abdel‐Khalek, A. M. , and Maltby J.. 2009. “Differences in Anxiety Scores of College Students From Germany, Spain, the United Kingdom, and the USA.” Psychological Reports 104, no. 2: 624–626. 10.2466/pr0.104.2.624-626. [DOI] [PubMed] [Google Scholar]
- Andrade, L. H. , Alonso J., Mneimneh Z., et al. 2014. “Barriers to Mental Health Treatment: Results From the WHO World Mental Health Surveys.” Psychological Medicine 44, no. 6: 1303–1317. 10.1017/s0033291713001943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Blatch Armon, D. , Buhayer A., Dobretz K., Meinlschmidt G., and Battegay E.. 2023. “Clinical Practice Guidelines for Cardiovascular Disease: How is Depression Addressed? Protocol for a Systematic Review.” BMJ Open 13, no. 5: e071940. 10.1136/bmjopen-2023-071940. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Casey, S. M. , Varela A., Marriott J. P., Coleman C. M., and Harlow B. L.. 2022. “The Influence of Diagnosed Mental Health Conditions and Symptoms of Depression and/or Anxiety on Suicide Ideation, Plan, and Attempt Among College Students: Findings From the Healthy Minds Study, 2018–2019.” Journal of Affective Disorders 298, no. Pt A: 464–471. 10.1016/j.jad.2021.11.006. [DOI] [PubMed] [Google Scholar]
- Cella, D. , Riley W., Stone A., et al. 2010. “The Patient‐Reported Outcomes Measurement Information System (PROMIS) Developed and Tested Its First Wave of Adult Self‐Reported Health Outcome Item Banks: 2005–2008.” Journal of Clinical Epidemiology 63, no. 11: 1179–1194. 10.1016/j.jclinepi.2010.04.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Choi, H. , Kim C., Ko H., and Park C. G.. 2019. “Translation and Validation of the Korean Version of PROMIS® Pediatric and Parent Proxy Measures for Emotional Distress.” Journal of Patient‐Reported Outcomes 3, no. 1: 36. 10.1186/s41687-019-0120-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Choi, S. W. , Gibbons L. E., and Crane P. K.. 2011. “Lordif: An R Package for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations.” Journal of Statistical Software 39, no. 8: 1–30. 10.18637/jss.v039.i08. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chon, K. K. , Suh K. H., Yi J., et al. 2023a. “Validation of the Korean Adaptation of the State‐Trait Anger Expression Inventory‐2 (K‐STAXI‐2).” Stress: The International Journal on the Biology of Stress 31, no. 3: 133–141. 10.17547/kjsr.2023.31.3.133. [DOI] [Google Scholar]
- Chon, K. K. , Suh K.‐H., Yi J., et al. 2023b. “The Korean Adaptation of the State‐Trait Anger Expression Inventory‐2: Initial Development.” Korean Journal of Health Psychology 28, no. 2: 537–559. [Google Scholar]
- Dong, L. , Li L., Wu Y., et al. 2024. “A Systematic Review of Interventions for Demoralization in Patients With Chronic Diseases.” International Journal of Behavioral Medicine. 10.1007/s12529-024-10262-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Elsman, E. B. M. , Flens G., de Beurs E., Roorda L. D., and Terwee C. B.. 2022. “Towards Standardization of Measuring Anxiety and Depression: Differential Item Functioning for Language and Dutch Reference Values of PROMIS Item Banks.” PLoS One 17, no. 8: e0273287. 10.1371/journal.pone.0273287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Elsman, E. B. M. , Roorda L. D., Crins M. H. P., Boers M., and Terwee C. B.. 2021. “Dutch Reference Values for the Patient‐Reported Outcomes Measurement Information System Scale v1.2‐Global Health (PROMIS‐GH).” J Patient Rep Outcomes 5, no. 1: 38. 10.1186/s41687-021-00314-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Embretson, S. E. , and Reise S. P.. 2013. Item Response Theory. Psychology Press. [Google Scholar]
- Flens, G. , Smits N., Terwee C. B., Dekker J., Huijbrechts I., and de Beurs E.. 2017. “Development of a Computer Adaptive Test for Depression Based on the Dutch‐Flemish Version of the PROMIS Item Bank.” Evaluation & the Health Professions 40, no. 1: 79–105. 10.1177/0163278716684168. [DOI] [PubMed] [Google Scholar]
- Gelkopf, M. , Mazor Y., and Roe D.. 2021. “A Systematic Review of Patient‐Reported Outcome Measurement (PROM) and Provider Assessment in Mental Health: Goals, Implementation, Setting, Measurement Characteristics and Barriers.” Supplement, International Journal for Quality in Health Care 34, no. S1: ii13–ii27. 10.1093/intqhc/mzz133. [DOI] [PubMed] [Google Scholar]
- Gondek, D. , Bernardi L., McElroy E., and Comolli C. L.. 2024. “Why Do Middle‐Aged Adults Report Worse Mental Health and Wellbeing Than Younger Adults? An Exploratory Network Analysis of the Swiss Household Panel Data.” Applied Research in Quality of Life 19, no. 4: 1459–1500. 10.1007/s11482-024-10274-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hofmann, S. G. , and Hinton D. E.. 2014. “Cross‐Cultural Aspects of Anxiety Disorders.” Current Psychiatry Reports 16, no. 6: 450. 10.1007/s11920-014-0450-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnsson, E. , Zolkowska K., and McNeil T. F.. 2015. “Prediction of Adaptation Difficulties by Country of Origin, Cumulate Psychosocial Stressors and Attitude toward Integrating: A Swedish Study of First‐Generation Immigrants From Somalia, Vietnam and China.” International Journal of Social Psychiatry 61, no. 2: 174–182. 10.1177/0020764014537639. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Monzonís‐Carda, I. , Adelantado‐Renau M., Beltran‐Valls M. R., and Moliner‐Urdiales D.. 2021. “An Examination of the Association Between Risk of Depression and Academic Performance According to Weight Status in Adolescents: DADOS Study.” Journal of Affective Disorders 290: 157–163. 10.1016/j.jad.2021.04.086. [DOI] [PubMed] [Google Scholar]
- Pilkonis, P. A. , Choi S. W., Reise S. P., Stover A. M., Riley W. T., and Cella D.. 2011. “Item Banks for Measuring Emotional Distress From the Patient‐Reported Outcomes Measurement Information System (PROMIS®): Depression, Anxiety, and Anger.” Assessment 18, no. 3: 263–283. 10.1177/1073191111411667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rodgers, M. , Dalton J., Harden M., Street A., Parker G., and Eastwood A.. 2018. “Integrated Care to Address the Physical Health Needs of People With Severe Mental Illness: A Mapping Review of the Recent Evidence on Barriers, Facilitators and Evaluations.” International Journal of Integrated Care 18, no. 1: 9. 10.5334/ijic.2605. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schalet, B. D. , Pilkonis P. A., Yu L., et al. 2016. “Clinical Validity of PROMIS Depression, Anxiety, and Anger Across Diverse Clinical Samples.” Journal of Clinical Epidemiology 73: 119–127. 10.1016/j.jclinepi.2015.08.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shadmi, E. , Gelkopf M., Garber‐Epstein P., Baloush‐Kleinman V., Doudai R., and Roe D.. 2018. “Routine Patient Reported Outcomes as Predictors of Psychiatric Rehospitalization.” Schizophrenia Research 192: 119–123. 10.1016/j.schres.2017.04.049. [DOI] [PubMed] [Google Scholar]
- Teresi, J. A. , and Fleishman J. A.. 2007. “Differential Item Functioning and Health Assessment.” Supplement, Quality of Life Research 16, no. S1: 33–42. 10.1007/s11136-007-9184-6. [DOI] [PubMed] [Google Scholar]
- Terwee, C. B. , Zuidgeest M., Vonkeman H. E., Cella D., Haverman L., and Roorda L. D.. 2021. “Common Patient‐Reported Outcomes Across ICHOM Standard Sets: The Potential Contribution of PROMIS®.” BMC Medical Informatics and Decision Making 21, no. 1: 259. 10.1186/s12911-021-01624-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Thornicroft, G. , and Slade M.. 2014. “New Trends in Assessing the Outcomes of Mental Health Interventions.” World Psychiatry 13, no. 2: 118–124. 10.1002/wps.20114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhang, Z. , Sun K., Jatchavala C., et al. 2019. “Overview of Stigma Against Psychiatric Illnesses and Advancements of Anti‐Stigma Activities in Six Asian Societies.” International Journal of Environmental Research and Public Health 17, no. 1: 280. 10.3390/ijerph17010280. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting Information S1
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.