The relationship between distress tolerance and symptoms of depression: Validation of the Distress Tolerance Scale (DTS) and short‐form (DTS‐SF)

Abstract

Objective: Distress tolerance (DT) has been found to be implicated in the development and maintenance of depressive symptomatology and various other significant psychological conditions. As such, it is critical to have measures of DT that are effective and easy to administer. This study aimed to examine the factor structure, psychometric properties, and clinical utility of the Distress Tolerance Scale (DTS) and the short‐form version, the Distress Tolerance Scale Short‐form (DTS‐SF), in a large population of individuals with varying levels of self‐reported depressive symptoms. Method: A total of 959 participants completed an online battery of questionnaires which included an assessment of depressive symptoms as well as the DTS and related measures. Results: Confirmatory factor analyses validated the four‐factor structure of the DTS and the one‐factor structure of the DTS‐SF. Good construct validity and good internal consistency were observed across both the DTS and DTS‐SF. Conclusion: Overall, this paper provides new evidence for the validity, reliability and discriminative ability of the DTS and the brief version of the questionnaire, the DTS‐SF.

Distress tolerance (DT) refers to the perceived capacity to endure and cope with uncomfortable or negative emotional states (Lass et al., 2020; Leyro et al., 2010; Simons & Gaher, 2005) ¹ . DT has been conceptualized as a trait‐like construct, with individual differences in the ability to tolerate distressing experiences (Simons & Gaher, 2005). Reduced DT has been associated with both poorer quality of life and reduced life satisfaction (Hu et al., 2014). Furthermore, DT has been implicated as a transdiagnostic risk/resilience factor in the onset and maintenance of various psychopathologies, including borderline personality disorder (Iverson et al., 2012), Post‐traumatic stress disorder (PTSD) (Vujanovic & Zegel, 2020), substance use disorder (Bornovalova et al., 2008), anxiety disorders (McHugh & Otto, 2012), and disordered eating (Juarascio et al., 2016). Indeed, research has demonstrated that low DT significantly predicts the use of maladaptive emotion regulation strategies including suppression of feelings, avoidance, rumination and substance use (Jeffries et al., 2016; Lass et al., 2020; Zegel et al., 2019). Similarly, poorer DT has been associated with an increased use of unhelpful self‐regulation strategies such as substance use, binge eating, or nonsuicidal self‐injury (Lass et al., 2020; McHugh & Otto, 2012). Given the significant transdiagnostic and mechanistic role of DT in various psychological conditions, research has examined treatments that attempt to improve an individual's DT (Brown et al., 2008; Leventhal & Zvolensky, 2015). The most well known of these is Dialectical Behavior Therapy (DBT; Linehan, 1993), which includes a specific module targeting DT deficits, and has been consistently demonstrated to improve outcomes (e.g., DeCou et al., 2019).

1. DT AND DEPRESSION

There is consensus that DT is a critical transdiagnostic construct relevant to affective disorders, impacting both intensity of experience, and coping strategy use (Allan et al., 2014; Lass & Winer, 2020; Michel et al., 2016). Depression is characterized by the experience of persistent negative emotions (e.g., low mood, hopelessness) and difficulties managing, appraising, and coping with such challenging emotions (Otte et al., 2016). Depression contributes substantially to the global burden of disease (Lépine & Briley, 2011), and is a commonly occurring comorbid condition that develops alongside other psychiatric disorders (Aina & Susma, 2006; Conner et al., 2009; Van & Kool, 2020), as well as chronic medical conditions (e.g., Nouwen et al., 2019). Further, depressed mood is associated with poorer quality of life, reduced social functioning, and poorer vocational functioning (Greenberg et al., 2003; Kupferberg et al., 2016).

Early cognitive models of depression (Abramson et al., 1989; Beck, 1987; Nolen‐Hoeksema, 1991) postulate that individuals with low mood exhibit alterations in their attention, interpretation, and processing of external and internal stimuli (Hankin et al., 2009). While these earlier theories focused on cognitive vulnerabilities, research has more recently investigated the role of individual's appraisals of their emotional experiences, including DT (i.e., an individual's perceived capacity to tolerate emotional distress) as a factor contributing to the development and maintenance of depressed mood states (Joormann & Stanton, 2016; Mehu & Scherer, 2015; Rottenberg, 2017; Yoon et al., 2018). Studies have consistently found that lower DT is associated with depressive symptoms (Allan et al., 2014; Beblo et al., 2012; Elhai et al., 2018; Leventhal & Zvolensky, 2015). Yoon and colleagues (2018) examined the relationship between individuals' attitudes towards negative emotions, which included their perceived capacity to tolerate distressing emotions (i.e., DT), and depression. Results of the meta‐analysis found that maladaptive attitudes to negative emotions (including poor DT) were correlated with greater severity of depressive symptoms. Yoon and colleagues (2018) concluded that this result demonstrated a systematic relationship between depression and poor DT.

Additionally, DT is implicated in the development and maintenance of a range of disorders including depression, PTSD (Zegel et al., 2019), eating disorders (Corstorphine et al., 2007), and adolescent alcohol use (Wahesh et al., 2020). Critically, unhelpful coping strategies (i.e., rumination and avoidance) have also been implicated as maintaining factors in depression (Felton et al., 2018; Thompson et al., 2010). Lass et al. (2020) used network analysis to explore the relationship between DT, depressive symptomology, and coping strategies. Results found a significant network relationship between depressive symptoms and low DT. Lass and colleagues concluded that DT interacts with an individual's motivation to engage or avoid challenging tasks, which in turn likely impacts the coping strategies (helpful or unhelpful) used during periods of distress, thus contributing to depressive experiences. Lastly, further evidence reinforcing the connection between DT and depressive symptoms comes from the treatment literature where two studies found individuals' DT had increased following an intervention for depression (McHugh et al., 2014; Williams et al., 2013). Moreover, Williams et al. (2013) demonstrated that as individuals' DT improved, depression symptoms reduced.

2. MEASURES OF DT

Given the significant impact of DT on depression and other disorders, it is imperative that researchers and clinicians have effective and efficient ways to measure this construct ² . Accordingly, Simons and Gaher (2005) developed the Distress Tolerance Scale (DTS), a 15 item self‐report measure. The DTS is comprised of four subscales that contribute to an overarching construct of DT (Simons & Gaher, 2005). Specifically, the four subscales are (1) appraisal and acceptability of emotions, that is, how an individual evaluates their negative emotions, and whether those emotions are acceptable, (2) the ability to tolerate negative emotions, (3) absorption, that is, how much an individual becomes absorbed in their distress and is consequently unable to complete other goals, and (4) regulation, that is the strategies an individual uses to regulate themselves (Simons & Gaher, 2005) ³ . Using exploratory and confirmatory factor analysis, Simons and Gaher (2005) validated the four‐facture structure of the DTS in a population of individuals with substance use disorder ⁴ . Subsequent research has since validated the four‐factor structure and the psychometric properties of the DTS, including test‐retest reliability, internal consistency, validity, and sensitivity to treatment effects, in general and clinical populations (e.g., Sandín et al., 2017; Leyro, et al., 2011). Further, whilst there have been studies assessing correlations between depression and low DT, to date there has been no psychometric investigation of the DTS in a depressed population. This is despite the research suggesting low levels of DT are implicated in the development and maintenance of depression (e.g., Gorka et al., 2012; Lass et al., 2020; Leventhal & Zvolensky, 2015).

The DTS offers a useful tool for the measurement of DT, yet there remains a need for brief measures, particularly in clinical contexts (Galesic & Bosnjak, 2009). Research has demonstrated that longer scales are associated with poorer quality of responding (Galesic & Bosnjak, 2009). To address this, Garner and colleagues (2018) developed a short‐form version of the Distress Tolerance Scale (DTS‐SF) composed of the four highest loading items from the four subscales of the original DTS. Validation of the measure was investigated in a sample of individuals with a diagnosis of OCD who underwent inpatient treatment (Garner et al., 2018). Garner and colleagues conducted an exploratory factor analysis, which supported a one‐factor structure for the four‐item DTS‐SF. In addition, the DTS‐SF correlated significantly with related measures and outcomes, such that lower DT was associated with symptom severity and reduced quality of life, and showed sensitivity to treatment effects, such that participant's scores improved significantly throughout the course of treatment (Garner et al., 2018). These results suggest the DTS‐SF is a promising brief measure of the construct. As yet, the current structure of the DTS‐SF has not been validated with a confirmatory factor analysis. Moreover, it has yet to be validated in a sample of individuals with depressive symptoms.

3. AIMS AND HYPOTHESES

The present study aims to assess the factor structure, factorial invariance, and psychometric properties of the DTS and the DTS‐SF in a large general sample of individuals with varying levels of self‐reported depressive symptoms. An additional goal of the study was to investigate the discriminative ability of both the DTS and the DTS‐SF, using receiver‐operating characteristic (ROC) curves to determine the sensitivity, specificity and predictive power of the two measures.

We hypothesized that the confirmatory factor analysis would support the four‐factor model of the DTS (Sandín et al., 2017; Simons & Gaher, 2005; Zvolensky, Leyro et al., 2011) and the one‐factor model of the DTS‐SF (Garner et al., 2018).

We expected good convergent validity would be observed between the DTS, DTS‐SF, and measures related to the construct of DT including the Difficulties in Emotion Regulation Scale (DERS; Conway et al., 2021; Gratz & Roemer, 2004), and the Depression Anxiety Stress Scales (DASS‐21; Lovibond & Lovibod, 1995) which is considered an indicator of not only depressive symptomatology but also overall psychological distress (Burton & Abbott, 2019; Henry & Crawford, 2005). Specifically, we anticipated scores on the DTS and DTS‐SF would correlate strongly, and negatively with these measures ⁵ . We also predicted the DTS and DTS‐SF would demonstrate evidence of reliability, through adequate internal consistency. Further, we hypothesized that DTS and DTS‐SF scores would differ significantly depending on self‐reported symptoms of depression, such that greater self‐reported depression would be associated with lower DTS and DTS‐SF scores. This hypothesis is based on extant literature demonstrating a strong relationship between DT and the experience of depression symptomatology.

4. METHOD

4.1. Participants

The total sample of 959 participants (60.6% female, mean age = 19.31 years, SD = 3.27 years, range = 17–56 years) included participants from two previously published studies (Burton et al., 2017, 2018). The sample comprised of first year psychology students at The University of Sydney who completed questionnaires in exchange for course credit.

Participants were grouped by their self‐reported depression scores on the Depression subscale of the DASS‐21, in keeping with previous research (Veilleux et al., 2019). Cut‐off scores for grouping were based on the DASS‐21 scoring information and severity cut‐offs resulting in four groups; normal, mild, moderate, and severe‐to‐extremely severe (Lovibond & Lovibond, 1995). As the number of participants who reported an extremely severe level of depression was quite small, the extremely severe and severe groups were merged (hereafter this group is referred to as severe+ in the paper). In total, participants who were categorized as “normal” on the DASS‐21 depression subscale comprised 505 participants (58.61% female, mean age = 19.40 years, SD = 3.66 years). A further 125 participants self‐reported as experiencing “mild” levels of depression (64.0% female, mean age = 19.45 years, SD = 3.30 years), 194 participants experienced “moderate” level of depression (63.40% female, mean age = 19.23 years, SD = 3.08 years), and 135 participants self‐rated experience of depression fell in the “severe+” range (60.74% female, mean age = 19.31 years, SD = 3.27 years).

Chi‐square tests of independence showed no significant relationship between gender and severity of depression, χ ²(6, 959) = 9.41, p = .15. A one‐way analysis of variance (ANOVA) found no significant difference in age across the different depression severity groups (normal, mild, moderate, severe/extremely severe), F(3, 955) = 0.66, p = .58.

4.2. Measures

4.2.1. The DTS

The DTS (Simons & Gaher, 2005) is a 15‐item self‐report scale that assesses an individual's ability to tolerate negative emotions. It consists of four subscales: appraisal of capacity and distress experienced (e.g., “I can tolerate being distressed or upset as well as most people), absorption in one's negative emotions (e.g., “When I feel distressed or upset, all I can think about is how bad I feel”), regulation of emotions (e.g., “I'll do anything to stop feeling distressed or upset), and tolerance of distressing emotions (e.g., “Feeling distressed or upset is unbearable to me”). Participants rate these items on a 5‐point Likert scale that ranges from strongly agree (1) to strongly disagree (5). The DTS is scored by calculating the mean score for each of the four subscales, and/or generating a mean total score (noting that item 6 requires reverse‐coding). When interpreting DTS scores, lower DTS scores indicating poorer ability to tolerate distress. The DTS demonstrated good test‐retest reliability (Simons & Gaher, 2005). In the current study, the DTS demonstrated good to excellent internal consistency in the current sample, α = .92 for the total score and α = .76–.86 for the DTS subscales.

4.2.2. The DTS‐SF

The DTS‐SF (Garner et al., 2018) ⁶ is a 4‐item self‐report measure that assesses DT. The DTS‐SF is a modification of the DTS and contains the highest factor loading item from each subscale, forming one DT factor. As with the DTS, items are rated on a 5‐point Likert scale (1 = strongly agree, 5 = strongly disagree), with lower scores suggesting poorer ability to withstand and tolerate distress. In Garner et al.'s (2018) study, the DTS‐SF demonstrated good test retest reliability. In the current study the DTS‐SF demonstrated adequate to good internal consistency across the whole sample, α = .81.

4.2.3. The DASS‐21

The DASS‐21 (Lovibond & Lovibond, 1995) is a 21‐item self‐report inventory used to measure the severity of depression (e.g., “I couldn't seem to experience any positive feeling at all”), anxiety (e.g., “I was aware of dryness of my mouth”), and stress (e.g., “I felt that I was using a lot of nervous energy”) that was experienced over the past week. Items are rated on a 4‐point Likert scale (0 “did not apply to me at all” to 3 “applied to me very much, or most of the time”). Lower scores indicate an absence or “normal” level of depression, anxiety, and stress symptoms. The DASS‐21 is a reliable measure commonly used to screen individuals for symptoms of depression and psychological distress. Participants who have higher total scores and higher depressive scores are associated with meeting diagnostic criteria for a Major Depressive Episode (Ng et al., 2007). The DASS‐21 has also been shown to correlate significantly with other measures of depressive symptoms like the Beck Depression Inventory‐II (Lovibond & Lovibond, 1995). The DASS‐21 total score provides a measure of general psychological distress (Burton & Abbott, 2019; Henry & Crawford, 2005). In the current study, the DASS‐21 total score demonstrated excellent internal consistency, α = .93, and good to excellent internal consistency was observed for the subscales, ranging from α = .80–.91.

4.2.4. The DERS

The DERS (Gratz & Roemer, 2004) is a 36‐item self‐report scale that measures an individual's ability to regulate their emotions. Participants rate each item according to how often they feel it applies to them from almost never (1) to almost always (6). The DERS is comprised of six subscales: lack of emotional awareness (e.g., “I pay attention to how I feel”), nonacceptance of emotional responses (e.g., “When I'm upset, I feel ashamed with myself for feeling that way”), impulse control difficulties (e.g., “When I'm upset, I lose control over my behaviors”), limited access to emotion regulation strategies (e.g., “When I'm upset, I believe that there is nothing I can do to make myself feel better), lack of emotional clarity (e.g., “I have no idea how I am feeling”), and difficulties engaging in goal‐directed behavior (e.g., “When I'm upset I have difficulty focusing on other things”). Higher scores on the DERS indicate greater difficulties in regulating emotions. Internal consistency was found to be excellent for the DERS total score α = .94, and good to excellent internal consistency for the subscales α = .80–.91.

4.3. Procedure

First year university students completed a battery of tests online using Qualtrics Survey Software which included demographic questions, the DASS‐21, the DTS, and the DERS. The research was approved by The University of Sydney Human Research Ethics Committee (Project Code: 2014_082) and all participants provided informed consent before participation.

4.4. Analyses

Analyses of the validity of the DTS and DTS‐SF, were conducted using SPSS, version 22 (IBM). Internal consistency was assessed using Cronbach's α as well as McDonald's ω, and Pearson's correlations were used to examine the construct and convergent validity. One‐way ANOVAs were used to examine between group differences. Confirmatory factor analyses (CFAs) and multigroup structural models were conducted using AMOS, version 12 to evaluate the four‐factor structure of the DTS and the one‐factor structure of the DTS‐SF. For all of the CFAs that were conducted, a maximum likelihood estimation was used with a maximum likelihood covariance matrix as input due to the tendency of the DTS to result in non‐normally distributed data (Hu & Bentler, 1999; Tonarely & Ehrenreich‐May, 2019). Multigroup CFAs were used to investigate whether the components of the four‐factor model of the DTS, and one‐factor model of the DTS‐SF were invariant (i.e., equivalent) across the different depression subgroups. Specifically, three levels of measurement invariance were tested: configural, metric, and scalar invariance. Configural invariance (unconstrained model) investigates whether the item‐factor structure of the model is the same across different groups. Metric invariance (weakly constrained model) investigates whether the item‐factor loadings are equivalent between the groups by forcing factor loadings to be equal and then comparing the change in model fit to the configural model. Lastly, scalar invariance (strongly constrained model) estimates whether the item intercepts are equivalent across the groups by making all intercepts equal across groups, and then comparing the change in model fit to the metric model. Partial scalar invariance was also investigated. Model fit was evaluated based off cut‐off values as suggested in the literature; comparative fit index (CFI), Tucker Lewis index (TLI), and incremental fit index (IFI) scores greater than 0.90 indicate acceptable fit (Hu & Bentler, 1999; Byrne, 1994). Root mean squared error of approximation (RMSEA) values of less than 0.6–0.8 represent good model fit (Hu & Bentler, 1999).

Discriminative ability was assessed with ROC curve analyses using MedCalc, version 19.4.1 (Medcalc Software). This assessment of the area under the curve (AUC) allows clinically significant scores to be distinguished from nonclinical scores. An AUC score of 0.70 or above is considered an acceptable discriminating score, whilst an AUC score below 0.65 is considered a weak classifier (Hosmer et al., 2013). MedCalc was also used to assess other markers of test performance including specificity (how likely are true positives), sensitivity (how likely are false negatives), positive predictive value (PPV; how likely the condition is present when the test is positive), and negative prediction value (probability the condition is not present when the test, i.e., the cut off score, is negative).

Complete DASS‐21 and DTS data were required for the analyses reported in this paper. In line with previous research (Morris & Coyle, 1994), if participants were missing one item per scale; the mean of the remaining items was substituted for the missing item score. If more than one item per subscale was missing, then the data was considered incomplete and omitted from analyses. This approach was used for all measures.

5. RESULTS

5.1. Confirmatory factor analysis and multigroup analysis

5.1.1. DTS

CFAs and multigroup CFA (were conducted in AMOS v12. The four‐factor solution (appraisal, absorption, regulation, and tolerance) was fitted to the data of the total sample, as well as the subgroup samples based on self‐reported depressive symptom severity (normal, mild, moderate, and severe+). In line with previous research, sample size was deemed adequate if the subjects to variable ratio were no lesser than five (Bryant & Yarnold, 1995; MacCallum et al., 1999), and all subgroups included in the multigroup analysis had a subjects‐to‐variables ratio greater than five (see Table 1 for specific sample sizes per analysis). The four‐factor solution was fitted to the entire sample and the results indicated acceptable fit. Refer to Table 2 for the standardized regression weights by sample group. Results of the series of CFAs across the subgroups identified acceptable fit to the normal and mild subgroup data and good fit to both the moderate subgroup and to the severe+ data (refer to Table 1). ⁷ These results support previous research (Simons & Gaher, 2005).

Table 1.

CFA fit statistics results for the DTS by self‐rated level of depressive symptom groups

	n	n per item	χ 2	df	χ 2/df	IFI	TLI	CFI	RMSEA (95% CI)
Total sample	959	63.94	547.37	84	6.52	0.94	0.91	0.94	0.07 (0.06–0.07)
Normal subgroup	505	33.00	366.29	84	4.36	0.92	0.89	0.92	0.08 (0.07–0.09)
Mild subgroup	125	8.30	157.55	84	1.88	0.91	0.89	0.91	0.08 (0.06–0.10)
Moderate subgroup	194	12.90	164.67	84	1.96	0.93	0.91	0.93	0.07 (0.05–0.09)
Severe+ subgroup	135	9.00	137.54	84	1.64	0.94	0.93	0.94	0.07 (0.05–0.09)

Note: N = 959; subgroups based on self‐reported depressive symptom severity; normal n = 505, mild n = 125, moderate n = 194, severe+ n = 135.

Abbreviations: CFA, confirmatory factor analyse; CFI, comparative fit index; CI, confidence interval; DTS, Distress Tolerance Scale.

Table 2.

Standardized regression weights of the DTS by self‐rated severity of depressive symptoms

DTS items	Total sample	Normal	Mild	Moderate	Severe+
Tolerance
Feeling distressed or upset is unbearable to me (1)	0.72	0.67	0.75	0.68	0.70
I can't handle feeling distressed or upset (3)	0.84	0.82	0.81	0.75	0.89
There's nothing worse than feeling distressed or upset (5)	0.68	0.65	0.62	0.65	0.72
Appraisal.
I can tolerate being distressed or upset as well as most people (6)	0.48	0.34	0.47	0.54	0.39
My feelings of distress or being upset are not acceptable (7)	0.67	0.68	0.48	0.52	0.57
Other people seem to be able to tolerate feeling distressed or upset better than I can (9)	0.75	0.68	0.65	0.76	0.76
Being distressed or upset is always a major ordeal for me (10)	0.82	0.77	0.88	0.80	0.74
I am ashamed of myself when I feel distressed or upset (11)	0.72	0.74	0.67	0.53	0.57
My feelings of distress or being upset scare me (12)	0.77	0.74	0.62	0.69	0.79
Absorption
When I feel distressed or upset, all I can think about is how bad I feel (2)	0.74	0.68	0.69	0.61	0.85
My feelings of distress are so intense they completely take over (4)	0.83	0.78	0.77	0.78	0.78
When I feel distressed or upset, I cannot help but concentrate on how bad the distress actually feels (15)	0.76	0.71	0.76	0.66	0.72
Regulation
I'll do anything to avoid feeling distressed or upset (8)	0.75	0.74	0.75	0.79	0.77
I'll do anything to stop feeling distressed or upset (13)	0.86	0.83	0.83	0.86	0.85
When I feel distressed or upset, I must do something about it immediately (14)	0.55	0.60	0.50	0.48	0.55

Note: N = 959; subgroups based on self‐reported depressive symptoms; normal n = 505, mild n = 125, moderate n = 194, severe+ n = 135.

Abbreviation: DTS, Distress Tolerance Scale.

The preceding CFA's demonstrated the four‐factor model had good fit in all depression groups (see Table 1), and thus is appropriate as a baseline model for the Multiple group CFA (Kyriazos, 2018). Next, a multigroup analysis was conducted with the data from each of the depression subgroups (normal, mild, moderate, and severe+). Criteria used to evaluate invariance included the change in CFI (ΔCFI ≤.01) and change in RMSEA (ΔRMSEA ≤0.015) (Chen, 2007; Kyriazos, 2018; Meade et al., 2008). An insignificant Δχ ² was not used as evaluation criterion, due to disparity between the sample sizes (Chen, 2007; Cheung & Rensvold, 2002). Results are presented in Table 3. The configural model showed acceptable fit (Chen, 2007), indicating that the four‐factor structure of the DTS was equivalent amongst the different groups. Metric invariance was then investigated. Results showed adequate model fit (CFI = 0.917, RMSEA = 0.38), and little difference between metric invariance model and the configural model (ΔCFI = 0.004, ΔRMSEA = 0.001). This result suggests the same latent factor is being measured in each group, and the factor loadings can be regarded as equivalent across the samples. As metric invariance was satisfied, we could proceed with testing scalar invariance. Results showed a reduction in the model fit (CFI = 0.867). Further, whilst the ΔRMSEA met criteria (ΔRMSEA = 0.008) the ΔCFI did not fulfill criteria (ΔCFI = 0.05). Thus scalar invariance was not met. This suggests the item intercepts could not be considered similar across the difference depression subgroups.

Table 3.

Fit statistics for the DTS multigroup CFA

	χ 2	df	χ2/df	IFI	TLI	CFI	RMSEA (95% CI)	ΔCFI	ΔRMSEA
Configural model	826.811	337	2.453	0.922	0.902	0.921	0.039 (0.036–0.042)
Metric model	884.043	370	2.389	0.918	0.906	0.917	0.038 (0.035–0.041)	0.004	0.001
Scalar model	1240.18	415	2.988	0.867	0.866	0.867	0.046 (0.043–0.049)	0.05	−0.008
Partial scalar modela	1158.35	403	2.874	0.978	0.873	0.878	0.044 (0.041–0.047)	0.039	−0.006

Abbreviations: CFA, confirmatory factor analyse; CI, confidence interval; DTS, Distress Tolerance Scale.

^a Four items were allowed to have free threshold parameters (items 4, 7, 14, and 15).

As this result precludes further analysis, investigation of the intercepts (i.e., partial scalar analysis) was conducted. Partial scalar analysis is considered an option to proceed with measurement invariance testing by relaxing the constraints on noninvariant item intercepts, (Jang et al., 2017; Putnick & Bornstein, 2016). Investigation of the intercepts revealed four intercepts with significant differences (items 4, 7, 14, 15). The model fit testing was repeated with these four intercepts allowed to be unconstrained. Although the partial scalar model found slightly improved model fit (CFI = 0.878), nevertheless it did not meet criteria for acceptable fit (Hu & Bentler, 1999). Comparing the partial scalar model to the metric model found ΔCFI >0.001, hence partial scalar invariance was not fulfilled. This suggests that whilst the underlying constructs and factorial structure is equivalent between the depression groups, the mean scores of the groups are not directly comparable (Allen et al., 2021).

5.1.2. DTS‐SF

A CFA of the DTS‐SF was then conducted in AMOS v12 with the total sample. As found by Garner and colleagues (2018) the four items comprising the DTS‐SF had the highest factor loadings on each subscale (also observed in our analysis, see Table 2). The one‐factor solution was fitted to the data, and demonstrated acceptable‐to‐moderate fit (see Table 4). Refer to Table 5 for the standardized regression weights by sample group. Next, a multigroup analysis was conducted with the data from each of the depression subgroups (see Table 6). The configural model found excellent fit for the one‐factor model of the DTS‐SF. The subsequent metric invariance model also demonstrated excellent fit, and resulted in little difference with the configural model (ΔCFI <0.01, ΔRMSEA <0.015). In contrast, the scalar invariance model resulted in a significantly poorer‐fitting model relative to the metric invariance model (CFI = 0.766, ΔCFI = 0.219, ΔRMSEA = −0.06), suggesting item intercepts vary significantly between different depression subgroups. Specifically, intercepts on items 4, and 10, were found to vary significantly. The model fit was substantially improved when these thresholds were allowed to have free parameters and approached significant fit (CFI = 0.885). Nevertheless, the partial scalar invariance was not satisfied as ΔCFI >0.01, and ΔRMSEA >0.015).

Table 4.

Fit statistics for the DTS‐SF, by self‐rated severity of depressive symptoms

Sample group	n	n per item	χ 2	df	χ2/df	IFI	TLI	CFI	RMSEA (95% CI)
Total sample	959	239.75	22.62	2	11.31	0.99	0.96	0.99	0.10 (0.07–0.14)
Normal subgroup	505	126.30	18.56	2	9.28	0.97	0.92	0.97	0.13 (0.08–0.18)
Mild subgroup	125	31.25	3.39	2	1.69	0.99	0.97	0.99	0.08 (0.00–0.21)
Moderate subgroup	194	48.50	2.97	2	1.49	0.99	0.98	0.99	0.05 (0.00–0.16)
Severe+ subgroup	135	33.75	1.47	2	0.74	1.00	1.01	1.00	0.00 (0.00–0.16)

Abbreviations: CFA, confirmatory factor analyse; CI, confidence interval; DTS‐SF, Distress Tolerance Scale short‐form.

Table 5.

Item factor loadings of the DTS‐SF by self‐rated severity of depressive symptoms

DTS‐SF items	Normal	Mild	Moderate	Severe+	Total sample
My feelings of distress are so intense that they completely take over	0.83	0.83	0.83	0.81	0.86
Being distressed or upset is always a major ordeal for me	0.82	0.83	0.82	0.81	0.85
I can't handle feeling distressed or upset.	0.85	0.80	0.76	0.80	0.84
I'll do anything to stop feeling distressed or upset.	0.58	0.67	0.64	0.62	0.65

Note: N  = 959; subgroups based on self‐reported depressive symptoms; normal n = 505, mild n = 125, moderate n = 194, severe+ n = 135.

Abbreviations: CFA, confirmatory factor analyse; CI, confidence interval; DTS‐SF, Distress Tolerance Scale short‐form.

Table 6.

Fit statistics for the DTS‐SF multigroup CFA

	χ 2	df	χ 2/df	IFI	TLI	CFI	RMSEA (95% CI)	ΔCFI	ΔRMSEA
Configural model	23.504	8	2.938	0.986	0.956	0.985	0.045 (0.025–0.067)
Metric model	32.589	17	1.917	0.985	0.979	0.985	0.031 (0.014–0.047)	0.00	0.014
Scalar model	277.993	29	9.586	0.765	0.806	0.766	0.095 (0.085–0.105)	0.219	−0.064
Partial scalar model	145.24	23	6.315	0.885	0.880	0.885	0.075 (0.063–0.086)	0.1	−0.044

Note: *Two items were allowed to have free threshold parameters (items 4, 10).

Abbreviations: CFA, confirmatory factor analyse; CI, confidence interval; DTS‐SF, Distress Tolerance Scale short‐form.

5.2. Reliability

5.2.1. Internal consistency

Cronbach's alphas and McDonald's omegas were calculated for the DTS total and the four DTS subscales for the total sample (N = 959), as well as for each depressive symptom severity subgroups, see Table 7. The DTS total was found to have good to excellent internal consistency and the DTS subscales were found to have adequate to good internal consistency. Of the four DTS subscales, the regulation subscale demonstrated the poorest internal consistency across both the total sample and the depressive symptom severity subgroups. Cronbach's alphas and McDonald's omegas were also calculated for the DTS‐SF which was found to have good internal consistency in the total sample and adequate‐to‐good internal consistency across the depressive symptom severity subgroups, see Table 7.

Table 7.

Internal consistency for the DTS‐SF and DTS subscales across groups

	DTS Total		DTS Tolerance		DTS Appraisal		DTS Absorption		DTS Regulation		DTS‐SF
Sample	α	ω	α	ω	α	ω	α	ω	α	ω	α	ω
Total sample	0.92	0.92	0.78	0.79	0.86	0.86	0.82	0.82	0.76	0.78	0.81	0.82
Normal subgroup	0.90	0.90	0.75	0.75	0.82	0.83	0.77	0.77	0.76	0.77	0.77	0.78
Mild subgroup	0.91	0.91	0.76	0.76	0.80	0.80	0.78	0.79	0.72	0.77	0.79	0.80
Moderate subgroup	0.89	0.89	0.72	0.73	0.81	0.81	0.72	0.73	0.74	0.77	0.77	0.78
Severe+ subgroup	0.91	0.91	0.80	0.82	0.80	0.80	0.82	0.82	0.76	0.78	0.76	0.77

Note: Subgroups based on self‐reported depressive symptoms; normal n = 505, mild n = 125, moderate n = 194, severe+ n = 135, total N = 959.

Abbreviations: CFA, confirmatory factor analyse; CI, confidence interval; DTS, Distress Tolerance Scale; DTS‐SF, Distress Tolerance Scale short‐form.

5.3. Test‐criterion validity

5.3.1. Group mean differences

DTS

Table 8 summarizes the mean total and subscale scores of the DTS for the sample grouped by self‐rated depressive symptom severity. A series of one‐way ANOVAs found significant differences between the groups. Specifically, there was a significant difference in DTS scores between individuals in the normal subgroup (n = 505; M = 51.86, SD = 11.07) compared with individuals who self‐reported experiencing any elevated level of depressive symptoms (from mild to severe/extremely severe symptoms (n = 454, M = 41.14, SD = 11.45). That is, individuals who self‐reported experiencing higher than “normal” depressive symptom severity scored significantly lower on the DTS than those reporting symptoms in the normal range (DTS total score: F(1, 957) = 217.36, p < .01). In addition, individuals who rated experiencing normal symptoms scored significantly higher on the DTS than participants who self‐reported experiencing depressive symptoms in the “mild” range (n = 125, M = 45.63, SD = 11.57), F(1, 628) = 31.24, p < .01). Participants who reported mild levels of depressive symptoms scored significantly higher on the DTS than individuals experiencing moderate levels of depressive symptoms (n = 194, M = 40.68, SD = 10.32), F(1, 317) = 15.86, p < .01). Last, there was also a small significant difference, between individuals reporting moderate levels of depressive symptoms compared to those reporting severe/extremely severe levels of depressive symptoms, with participants who fell in severe/extremely severe range for depressive symptoms showing significantly lower DTS scores (n = 135, M = 37.63, SD = 11.60), F(1, 327) = 6.27, p = .01).

Table 8.

DTS subscale and DTS‐SF means (and SDs) by groups

Depression group	DTS total score	DTS Tolerance Scale	DTS Absorption Scale	DTS Appraisal Scale	DTS Regulation Scale	DTS‐SF
Normal	51.87 (11.07)	9.79 (2.88)	10.28 (2.82)	22.42 (4.86)	9.38 (2.79)	14.33 (3.50)
Mild	45.63 (11.58)	8.66 (2.92)	8.42 (2.86)	19.95 (5.06)	8.61 (2.71)	12.22 (3.80)
Moderate	40.69 (10.32)	7.82 (2.63)	7.29 (2.54)	17.25 (4.83)	8.32 (2.50)	10.93 (3.38)
Severe+	37.64 (11.60)	7.25 (3.01)	6.59 (2.89)	15.56 (5.06)	8.23 (2.80)	10.01 (3.59)

Note: Subgroups based on self‐reported depressive symptoms; normal n = 505, mild n = 125, moderate n = 194, severe+ n = 135.

Abbreviations: DTS, Distress Tolerance Scale; DTS‐SF, Distress Tolerance Scale short‐form.

DTS‐SF

Table 8 summarizes the mean scores of the DTS‐SF by self‐rated depressive symptom severity. As with the DTS, individuals who scored in the normal range (n = 505, M = 14.33, SD = 3.49) scored significantly higher on the DTS‐SF than participants who scored in the mild to severe/extremely severe range for depressive symptoms (n = 454, M = 11.01, SD = 3.65), F(1, 957) = 206.14, p < .01). Significant differences on scores on the DTS‐SF were also found between self‐classified levels of depression. Individuals in the normal subgroup scored significantly higher on the DTS‐SF than participants in the “mild” subgroup (n = 125, M = 12.22, SD = 3.80), F(1, 628) = 35.12, p < .01). Similarly, individuals reporting mild levels of depressive symptoms scored significantly higher on the DTS‐SF, than individuals reporting moderate levels of depressive symtpoms (n = 194, M = 10.93, SD = 3.60), F(1, 317) = 10.05, p < .01). Finally, there was a small but significant difference between individuals experiencing moderate depressive symptoms, and individuals reporting severe/extremely severe depressive symptoms, with participants who fell in severe+ range having significantly lower scores on the DTS (n = 135, M = 10.01, SD = 3.50), F(1, 327) = 5.59, p = .02).

5.3.2. Correlations

Convergent validity between the DTS total and subscales and the DTS‐SF was assessed by examining the correlations between these scales and measures assessing similar constructs, including the DERS and DASS‐21 (see Table 9).

Table 9.

Correlations between DTS total and subscales and DTS‐SF, with the DERS total and subscales, and the DASS‐21

Scale	DTS Total	DTS Tolerance	DTS Appraisal	DTS Absorption	DTS Regulation	DTS‐SF Total
DTS Total	–	0.85**	0.91**	0.88**	0.71**	0.95**
DTS Tolerance	0.85**	–	0.66**	0.76**	0.53**	0.84**
DTS Appraisal	0.91**	0.66**	–	0.75**	0.51**	0.84**
DTS Absorption	0.88**	0.76**	0.75**	–	0.49**	0.87**
DTS Regulation	0.71**	0.53**	0.51**	0.49**	–	0.66**
DTS‐SF	0.95**	0.84**	0.84**	0.87**	0.66**	–
DASS‐21 Total	−0.59**	−0.50**	−0.60**	−0.57**	−0.28**	−0.57**
DERS Total	−0.70**	−0.53**	−0.74**	−0.65**	−0.35**	−0.66**
DERS: Nonacceptance	−0.58**	−0.43**	−0.64**	−0.47**	−0.32**	−0.51**
DERS: Goals	−0.57**	−0.48**	−0.53**	−0.61**	−0.30**	−0.56**
DERS: Impulse	−0.63**	−0.48**	−0.64**	−0.59**	−0.35**	−0.60**
DERS: Awareness	−0.06	−0.01	−0.14**	0.02	0.02	−0.04
DERS: Strategies	−0.68**	−0.52**	−0.69**	−0.68**	0.65**	−0.66**
DERS: Clarity	−0.40**	−0.28**	−0.45**	−0.36**	−0.18**	−38**

Abbreviations: DASS, Depression Anxiety Stress Scales; DTS, Distress Tolerance Scale; DTS‐SF, Distress Tolerance Scale short‐form.

^**Correlation is significant at the 0.01 level (two‐tailed).

The DTS subscales were found to correlate significantly with each other, falling in the moderate range (r = .49–.76). Correlations between the DTS total score and DERS total and subscales, and DASS‐21 were significant and negatively correlated, falling in the moderate range (r = −.40–.68), with the exception of the DERS Awareness subscale, which was nonsignificantly correlated with DTS‐SF, DTS total, as well as the DTS Tolerance and Absorption subscales.

Correlations between the DTS‐SF, the DTS total and subscales, DASS total score, DERS total and subscales were significant and negatively correlated, ranging from weak to strong correlations (r = −.18–.74), see Table 9.

5.4. ROC curve

To assess the discriminative ability of the DTS and DTS‐SF, ROC curve analyses were performed. Table 10 presents the Youden Criterion score, sensitivity, specificity, PPV, NPV, and AUC results for curves assessing the measures' ability to discriminate between “normal” (n = 505) and “depressive symptoms” (n = 454) subjects in the sample (the sample group “depressed” combines the mild, moderate and severe+ sample subgroups). Analyses found that a total score of 47 or less in the DTS could reliably discriminate between respondents from the “normal” and the “depressive symptoms” subgroups and a score of 11 or less on the DTS‐SF could discriminate respondents from the “normal” and the “depressive symptoms” subgroups. See Table 10 for further details of the ROC curve analysis.

Table 10.

Results from receiver operating characteristic curve (ROC) analysis for the DTS and DTS‐SF

	Cut‐Off Score	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	AUC (95% CI)
DTS Total Score (Normal vs. Depressive Symptoms)	≤47	73.13 (68.80–77.20)	65.54 (61.20–69.70)	65.60 (62.60–68.50)	73.10 (69.70–76.20)	0.75** (0.73–0.78)
DTS‐SF (Normal vs. Depressive Symptoms)	≤11	75.77 (71.60–79.60)	60.59 (56.20–64.29)	63.40 (60.50–66.10)	73.60 (70.00–76.90)	0.75** (0.72–0.77)

Abbreviations: CI, confidence interval; DTS, Distress Tolerance Scale; DTS‐SF, Distress Tolerance Scale short‐form; PPV, positive predictive value.

6. DISCUSSION

DT has increasingly been investigated as a key risk/resilience factor in the development and maintenance of depression (Lass et al., 2020). As such, it is important to have useful and reliable tools for the measurement of DT. This study aimed to validate the factor structure, multigroup factorial invariance, psychometric properties, and clinical utility of the DTS and the DTS‐SF in a sample of individuals with varying levels of self‐reported depressive symptoms. As hypothesized, confirmatory factor analysis validated the four‐factor structure of the DTS, and the one‐factor solution of the DTS‐SF. Configural and metric invariance was found for the DTS, and DTS‐SF, although scalar and partial scalar invariance was not fulfilled. Moreover, as predicted the DTS and DTS‐SF correlated significantly with related measures, the DERS, and DASS‐21, demonstrating good convergent validity. Further, results from the ROC curve analyses provided evidence of discriminative ability for the DTS and DTS‐SF. Overall, the results of this study indicate that the DTS and DTS‐SF are valid, clinically useful measures that capture DT. These findings build upon the previous research that has demonstrated a significant relationship between DT and depressive symptoms (Yoon et al., 2018). The findings of this study extend existing research by validating the factor structure of the DTS‐SF, a promising brief measure.

6.1. CFA and multigroup CFA factor structure

As predicted, the confirmatory factor analysis validated the four‐factor structure of the DTS, with moderate to good fit to the data of the total sample, as well as in each of the subgroups based on self‐reported depressive symptom severity. This replicates the factor structure observed in Simons and Gaher's (2005) original analyses, as well as subsequent examinations of the measure in various clinical populations (Azizi, 2010; Leyro et al., 2011; Sandín et al., 2017). It was observed that the strength of the fit statistics increased with the severity of depression, such that DTS demonstrated the best fit to the data in the severe+ group. Overall, these results further contribute to our understanding of the relationship between DT and depressive symptomatology.

Also as predicted, the results of the confirmatory factor analysis validated the one‐factor structure of the DTS‐SF, with good fit to the data of the total sample and in each of the subgroups based on self‐reported depressive symptom severity. This replicates Garner and colleagues (2018) findings of a one‐factor structure from their exploratory analysis. Moreover, these results are in keeping with previous research demonstrating the significant relationship between depressive symptoms and poor DT (Allan et al., 2014; Lass & Winer, 2020; Leventhal & Zvolensky, 2015; Yoon et al., 2018). Notably, the best fit was observed for the severe+ group. This study also replicated Garner et al.'s (2018) finding that the four items for the DTS‐SF were the highest factor loadings from each of four subscales (appraisal, tolerance, absorption, and regulation) of the original 15‐item DTS. The factor loadings were also highest for all four items in the severe+ group. Amongst the moderate, mild, and normal levels of self‐reported depressive symptoms groups, the fourth item “I will do anything to stop feeling distressed or upset” demonstrated a relatively low factor loading compared with the other three items. This finding also aligns with that of Garner et al. (2018) who proposed that this item may be capturing a related construct such as unhelpful coping strategies, or difficulties in emotion regulation. Given our findings also found a similar pattern, further research investigating the relationship to other scales assessing those constructs is recommended, as previous research has also demonstrated the regulation subscale in the DTS has a strong overlap with emotion regulation constructs and associated measures (Leyro et al., 2010).

Multigroup confirmatory factor analysis enables evaluation of the psychometric equality of constructs between groups (Putnick & Bornstein, 2016), and thus the generalizability of the construct. Investigation of measurement invariance of the four‐factor DTS, and one factor DTS‐SF across the four subgroups based on self‐reported severity of depressive symptoms (normal, mild, moderate, severe+) supported configural and metric invariance. That is, our results indicate the DTS and DTS‐SF have a similar underlying factor structure, and equivalent factor loadings for all depressive symptoms subgroups. However, neither scalar nor partial scalar invariance were fulfilled for either scale, suggesting that the item intercepts (and hence group means) vary between these four groups. (Given the same sample group for the DTS and DTS‐SF, it is unsurprising to see a similar pattern of scalar and partial scalar not being achieved for either measure). This disparity between item intercepts may be due to one of the depressive symptoms subgroups methodically rating some items higher or lower than other groups. The item that varied most in the DTS and DTS‐SF was item four “my feelings of distress are so intense they completely take over.” Individuals' in the severe+ group were more likely to rate that as “strongly agree (1)” compared with the normal group who responded most with “disagree/strongly disagree.” Research has shown that there is a significant difference in the experience and level of distress experienced between individuals with depression, compared to nonclinical population (Richards, 2011; Rottenberg, 2017). Furthermore, it has been shown that substantial differences exist between individual's with and without depressive symptoms in the metacognitive beliefs about their own distress and capacity to tolerate it (Chen et al., 2021; Liu & Thompson, 2017; Salmani & Hasani, 2016). Moreover, the experience of depressive symptoms and depression has been shown to modify individual's perceptions of their capacity to cope with distress and DT (Beblo et al., 2012; Werner‐Seidler et al., 2013). Thus, given the role of DT in depression it may explain why the DTS and DTS‐SF do not function unvaryingly between different subgroups that differ in depression symptomatology. This result of variance is also in line with previous research, whereby nonscalar invariance is not uncommon between clinical and nonclinical populations, particularly for scales that assess specific psychological constructs (Costa et al., 2014; Iliadis et al., 2020; Ito et al., 2015). Overall, these results provide useful information regarding the factor structure, and generalizability of the DTS and DTS‐SF in individuals with varying levels of depression. Future research should continue to investigate the multigroup invariance of these two measures in clinical populations of individuals with a diagnosis of depression and/or other psychological disorders.

6.2. Reliability, validity, and clinical utility

As anticipated, DTS total, subscale scores, and DTS‐SF scores significantly differed depending on the severity of depression, such that scores were significantly lower for individuals who self‐reported experiencing increased symptoms of depression. Moreover, significant differences in performance on these measures was also found between self‐classified groups of depression; individuals who rated themselves as experiencing severe to extremely severe levels of depression had lower scores on the DTS and DTS‐SF (i.e., poorer DT capacity) than individuals who rated themselves as experiencing mild or moderate levels of depressive symptoms. Overall, this indicated individuals with high levels of depressive symptoms perceive themselves to have diminished DT.

In keeping with our hypotheses, our results indicated good internal consistency (α > .70) across the different groups of self‐rated depressive symptoms for both the DTS and DTS‐SF and met the Terwee criteria for adequate internal consistency (Terwee et al., 2007). Further, the DTS total score and its four subscales and the DTS‐SF were significantly and negatively correlated with a measure of general psychological distress (DASS‐21 total score; Burton & Abbott, 2019; Henry & Crawford, 2005) and a related measure of emotion regulation, the DERS. This provides further evidence for convergent validity of the DTS and DTS‐SF and reinforces past research showing a relationship with low DT, higher symptoms of depression, and poor emotion regulation strategies (Hu et al., 2014; Leyro et al., 2011).

There was a significant, strong positive correlation between the DTS total and subscale scores, and the short form, reinforcing that the DTS‐SF is capturing the construct DT. All correlations between the related measure of the DERS fell in the moderate range, with the exception of the subscale of Awareness. This nonsignificant finding may be due to the Awareness measure being the least valid and reliable subscale in the DERS questionnaire (Bardeen et al., 2013; Cho & Hong, 2013). Indeed, the developers of the DERS‐16, an abbreviated version of the DERS (Bjureberg et al., 2016) excluded the Awareness subscale due to the lack of fit, although other investigations have found support for this subscale (Victor & Klonsky, 2016). Overall, these results are consistent with previous evidence demonstrating diminished DT correlates with increased depressive symptoms (Lass et al., 2020) and poorer emotion regulation strategies (Thompson et al., 2010).

Finally, as predicted, ROC curve analyses identified good discriminative ability for the DTS total score and DTS‐SF, with all scores having significant AUC values (Hosmer et al., 2013). Future research could use ROC curve analysis to investigate potential clinical cut‐off scores for the DTS and DTS‐SF in relevant clinical samples such as borderline personality disorder, substance use disorders, and affective disorders.

6.3. Theoretical implications

Our results provide strong evidence that poor DT is significantly associated with depressive symptomatology, consistent with existing research (Yoon et al., 2018). These results reinforce the importance of integrating the role of DT into existing models of depression. To date, only a relatively small number of studies has examined and incorporated DT or related constructs in models of the development of depression. For example, Rottenberg (2017) investigated the role of individuals' emotion regulation strategies (a construct related to DT) in depression and results highlighted an important role emotion regulation plays in the development of depression. Furthermore, a small number of studies have integrated DT into the stress‐diathesis models of depression (Felton et al., 2018; Macatee et al., 2016). The stress‐diathesis model suggests that depression and low mood can develop through interactions between stressful life events and pre‐existing vulnerabilities (referred to as diatheses) (Paredes & Zumalde, 2015). Felton et al. (2018) proposed that an individual's DT may be a key diathesis that could predispose an individual to coping poorly with a stressful life event, and subsequently developing low mood. In keeping with this model, Felton and colleagues (2018) found that adolescents who had poor DT were more likely to develop depressive symptoms after experiencing stressful life events, than adolescents who experienced similar stressful events yet had higher DT. This is further supported by Lass and colleagues’ (2020) findings on the interaction between DT, depressive symptoms, and unhelpful coping strategies. Future research should continue to investigate the relationship between poor DT and depression and integrate this into current models.

6.4. Clinical implications

DT has been shown to improve in individuals receiving treatment for depression (Williams et al., 2013), yet it is unclear which aspect of treatment impacts DT, or if DT improves in general throughout evidence‐based treatment for depression. Consequently, accurate measurement of the DT construct is also critical. Moreover, research has demonstrated the usefulness of regularly monitoring progress throughout treatment. Although multiple measures of DT exist, notably, all scales are long (Garner et al., 2018), which impedes the ability as an effective tracking questionnaire. Research has highlighted the length of a scale impacts the quality of responding, hence the need for brief measures (Galesic & Bosnjak, 2009). Additionally, brief measures play an important role as screening tools. With only four items, the DTS‐SF has great potential to be used as a screening measure to identify diminished DT. If a respondent's scores indicate significantly low DT then further assessment can be conducted, and the use of specific, longer questionnaires like the full‐scale DTS or DERS would be both appropriate and helpful. Thus, these results of the DTS‐SF as a valid, brief measure have important clinical implications and enable easy use for the purpose of symptom screening and tracking.

6.5. Limitations and future directions

A major limitation of this study is that it did not utilize a clinical sample. In the current study, depressive symptoms in a large general undergraduate sample were assessed by self‐report questionnaire, in a cross‐sectional design. Future research should endeavor to assess the factor structure and measurement invariance in a population of clinically diagnosed individuals with Major Depressive Disorder (MDD). Further, it would be useful to explore whether any differences in DT exist between individuals experiencing a single episode of MDD, and individuals experiencing recurrent episodes or chronic depression (i.e., persistent depressive disorder), as chronic depression has been shown to have higher levels of dysfunctional coping compared with nonchronic MDD (Schramm et al., 2020), and dysfunctional coping is strongly associated with DT (Lass et al., 2020). Also, given the potential higher‐order factor of DT (as identified by Simons and Gaher (2005) and indicated in the present study by the strong inter‐factor correlations for the DTS), another avenue for future research would be to further assess and validate the factor structure of the DTS using exploratory structural equation modeling as an alternative to CFAs.

Another limitation was that this paper was unable to assess divergent validity. We only assessed convergent validity through examining the correlations between the DTS, DTS‐SF, and the related measure the DERS (Gratz & Roemer, 2004), and the DASS‐21 (Lovibond & Lovibond, 1995). Future research would be enhanced by a nomological network study of the construct of DT. Additionally this study was unable to ascertain test‐retest reliability or treatment sensitivity of the DTS or DTS‐SF with a depressed sample, which should be addressed in future research. Similarly, examining responsiveness to treatment in clinical population receiving treatment for depression will not only provide further necessary psychometric information about the scale, but simultaneously elucidate further information about the role DT plays in the maintenance of depressive symptoms. Thus, future research could evaluate the impact of different treatments targeting DT (e.g., such as DBT) in individual's presenting for treatment for depression. Lastly, as a psychometric study, this paper is unable to answer questions about the causal connection between depression and DT.

In summary, this study provides new evidence of the validity, clinical utility, and factor structure and factorial invariance of the DTS and the DTS‐SF in a population of individuals with varying levels of self‐reported depressive symptoms. Importantly, our findings suggest that the DTS‐SF is a promising brief measure of DT, which has great potential application in clinical settings. The role of DT as a key maintaining factor in depression deserves ongoing theoretical and empirical attention. The DTS and DTS‐SF will be useful in such future endeavors.

AUTHOR CONTRIBUTIONS

All authors contributed to the study conception and design. Material preparation and data collection were performed by Dr Amy Burton and analysis was performed by all authors. The first draft of the manuscript was written by Ruby Brown and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

PEER REVIEW

The peer review history for this article is available at https://publons.com/publon/10.1002/jclp.23370

ETHICS STATEMENTS

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Human Research Ethics Committee of The University of Sydney (Project Code: 2014_082).

Brown, R. J. , Burton, A. L. , & Abbott, M. J. (2022). The Relationship Between Distress Tolerance and Symptoms of Depression: Validation of the Distress Tolerance Scale (DTS) and short‐form (DTS‐SF). Journal of Clinical Psychology, 78, 2609–2630. 10.1002/jclp.23370

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon request.