Screening Young Adult Cancer Survivors for Distress with the Distress Thermometer (DT): Comparisons with a Structured Clinical Diagnostic Interview

Abstract

BACKGROUND

The validity of the Distress Thermometer (DT) as a screen for psychological distress in young adult (YA) cancer survivors was assessed by comparing it with results of a psychiatric diagnostic interview, the Structured Clinical Interview for the DSM-IV (SCID), in order to evaluate accuracy of the DT and identify optimal cut-off scores for this population.

METHODS

247 survivors (age 18–40 years) completed the DT and SCID. Based on the SCID, participants were classified as having: 1) One or more SCID diagnoses; 2) Significant symptoms, but no SCID diagnosis; or 3) No significant SCID symptoms. ROC analyses determined sensitivity and specificity of all possible DT cut-off scores for detecting survivors with a SCID diagnosis, and subsequently for survivors with significant SCID symptoms or a SCID diagnosis.

RESULTS

The recommended DT cut-off score of ≥ 5 failed to identify 31.81% of survivors with a SCID diagnosis (sensitivity 68.18%, specificity 78.33%), and 32.81% of survivors with either Significant SCID symptoms or a SCID diagnosis. No alternative DT cut-off score met criteria for acceptable sensitivity (≥.85) and specificity (≥.75).

CONCLUSIONS

The DT does not reliably identify YA cancer survivors with psychiatric problems identified by a “gold standard” structured psychiatric interview; the DT should not be used as a stand-alone psychological screen in this population.

After completion of treatment, most cancer survivors adapt well and will not experience significant long-term psychological distress.^1–3 However, as a group, cancer survivors are at increased risk for psychological maladjustment, including depression, anxiety, and suicide, even many years after completion of therapy.^1–7 Because of this risk, psychological screening has been widely advocated for cancer survivors,^8,9 as reflected in treatment guidelines for post-treatment care.¹⁰ Unfortunately, most guidelines offer limited guidance about how recommended psychological screening of survivors should be conducted. Though a variety of screening measures have been introduced, few studies have specifically examined the validity of distress screening measures in cancer survivors, and even fewer have compared screening measures to diagnostic interview measures commonly used in psychiatric assessment research.^11–13 This is important because without comparisons to “gold standard” diagnostic interviews, it is difficult to determine how accurately these measures identify significant mental health problems.

Young adults (YA) are thought to be a particularly vulnerable survivor group, as cancer may disrupt their physical and emotional development, and they may live with late-effects for many years after treatment.^14,15 Several studies of YA survivors show elevated rates of psychological distress^1,4 and barriers to receiving appropriate psychosocial care.¹⁶ Reviews of adolescent and YA survivor care note that both research and patient care would be significantly improved by patient-reported outcome measures specifically validated for this population.^15–17

To address this need, we set out to evaluate the Distress Thermometer (DT)^12,18 as a psychological screening measure for YA cancer survivors. The DT is a single-item self-report method for identifying psychological distress that has been widely recommended for screening cancer patients^19,20 and survivors.²¹ Several studies have reported favorably on the performance of the DT relative to other self-report measures of anxiety and depression symptoms. ^12,22,23 However, there have been questions about whether a DT cut-off score of 4 or 5 is optimal for distress screening cancer patients.¹² In addition, some studies have raised significant questions about how well the DT identifies individuals with significant psychological distress.^24,25 Specifically, some studies report the DT fails to identify a significant proportion of individuals with significant psychological symptoms reported on other measures.^25–27 Prior studies of the DT have almost exclusively compared its performance to symptom checklist measures and have been limited to patients during or shortly after their treatment. By evaluating the DT in a sample of YA cancer survivors and comparing it to a structured diagnostic interview, this study aimed to evaluate how accurately the DT would identify individuals with psychological distress and to identify which cut-off score would be optimal for this population.

Method

Participants

Participants were locally-followed YA cancer survivors recruited primarily at an oncology follow-up visit at our cancer center. Participants were recruited from six disease centers providing cancer treatment and aftercare (Pediatric Oncology, Breast Oncology, Hematological Oncology, Genitourinary Oncology, Adult Neurooncology, and Sarcoma Clinic) and three long-term follow up clinics (Pediatric Survivorship Clinic, Pediatric Neuro-Outcomes Clinic, and Adult Survivorship Clinic. In addition, 2 participants were recruited at an education program for young breast cancer patients.

To be eligible, survivors needed to be age 18–40, at least three years from cancer diagnosis, at least two years since treatment completion (excluding chemoprevention), and able to complete self-report measures in English. Eligibility criteria were assessed by medical record review and verified by patient report. To encourage candid responses from participants, once enrolled, study information was collected anonymously without links to personal identifiers or medical records. We planned to enroll equal numbers of survivors diagnosed before and after age 21, with each group stratified by gender. During the study period, 349 eligible survivors were approached and 250 (71.6%) consented and enrolled. Study procedures were approved by the cancer center’s institutional review board.

Measures

Participants completed all measures during a single study visit

Demographic and Treatment Information

Participants reported their demographic and treatment information on a study questionnaire.

Distress Thermometer (DT)

The DT is a one-item self-reort screening tool for measuring psychological distress in cancer patients.^12,18 The DT measures distress levels over the prior week using a visual analogue scale with scores from 0 (no distress) to 10 (extreme distress). The DT has been used in various populations and its performance compared with a variety of self-report symptom scales.^12,20,22,23 Scores of 4 and 5 have each been recommended as potential cut-off scores on the DT to indicate significant distress.^12,19,22,23 Similar to previous DT studies,^12,20,22,23 only the visual analogue DT itself was used; the companion problem list was omitted.

Structured Clinical Interview for the DSM-IV (SCID)

The SCID, a semi-structured clinical interview for making psychiatric diagnoses based on DSM-IV criteria, is one of the most widely accepted methods for making psychiatric diagnoses in research.^28–30 The SCID uses standard questions and follow-up probes to elicit respondents’ descriptions of their psychiatric symptoms. Items are keyed to decision trees for psychiatric diagnoses, and skip patterns in the interview insure respondents who fail to meet minimal criteria for a diagnosis are not administered additional items.

The research version of the SCID, used here, is designed to be tailored by selecting relevant diagnostic modules. Depressive and anxiety disorder modules were selected for the study because these are common psychological concerns of cancer survivors.^1,3,5,31 Diagnostic modules included were: Major Depressive Disorder, Minor Depressive Disorder, Dysthymia, Panic Disorder (with and without Agoraphobia), Agoraphobia, Specific Phobia, Social Phobia, Obsessive-Compulsive Disorder, Generalized Anxiety Disorder, Acute Stress Disorder, Mixed Anxiety and Depressive Disorder, and Adjustment Disorder. Of note, we included only screening items for Post-Traumatic Stress Disorder (PTSD), asking respondents to describe any past traumatic event and current associated symptoms, because distress measures like the DT evaluate the presence of current symptoms, and not symptom etiology to discern if symptoms are related to a past traumatic event. Participants endorsing these symptoms were categorized as having anxiety symptoms, but not diagnosed with PTSD. Only current symptoms and diagnoses were scored, defined by the SCID as present in the past 30 days. ^30,32

All SCID interviews were conducted by a single interviewer (JB) blind to subjects’ DT responses. The SCID screening module was adapted by placing initial critical items for each diagnostic module on the screener. Each participant was administered all these critical items (Supplemental Material 1) and those who endorsed an item were subsequently administered relevant diagnostic modules. We administered and scored each SCID item using standard SCID response categories. SCID interviews were scored using standard algorithms for DSM-IV-TR diagnoses.

Statistical Analysis

Based on the SCID, participants were classified into one of three categories: 1) SCID Diagnosis, including participants who met criteria for at least one SCID diagnosis; 2) Significant SCID Symptoms, including participants who endorsed two or more critical symptoms (Supplemental Material 1), but did not meet criteria for a SCID diagnosis; and 3) No Significant SCID Symptoms, including survivors who endorsed fewer than two critical symptoms and did not meet criteria for a SCID diagnosis. Descriptive statistics were used to describe the distribution of the sample across these three categories and across the range of DT scores.

ROC analyses first examined the DT’s discrimination between survivors with and without a diagnosis, and subsequently between survivors with No Significant SCID symptoms, compared to survivors in the Significant SCID Symptoms or SCID Diagnosis groups. Analyses were conducted first for all diagnoses and symptoms combined, and subsequently for anxiety and depression separately. Area under the ROC curve (AUC), and sensitivity and specificity for all possible cut-off scores of the DT were calculated. AUC values ≥ 0.80 were taken to reflect good discrimination and values ≥ 0.90 to reflect excellent discrimination. To insure that most, if not all, affected patients are identified, clinical screening programs typically select screening criteria with high sensitivity.^33,34 Based on previous studies demonstrating psychiatric screening instruments can demonstrate high sensitivity and maintain robust specificity,^25,34–36 we determined a priori that a DT cut-off score with sensitivity ≥ 85% and specificity ≥ 75% would be required for clinical screening purposes.

Results

Sample Description and Distribution on the SCID and DT

Of the 250 participants enrolled, 247 completed the DT, and were included here. Participants were 124 males (62 diagnosed < age 21) and 123 females (63 diagnosed age < 21). Mean age at enrollment was 29.4 years (SD=7.37). Age of first cancer diagnosis ranged from birth to 37 years (mean=20.41 years, SD=9.78) (Table 1).

Table 1.

Description of the Sample (N= 247)

Gender	N	%
Male	124	50.2
Female	123	49.8
Ethnicity
Caucasian	206	83.4
Hispanic	14	5.7
Asian/Pacific Islander	7	2.8
African-American	6	2.4
Other	11	4.5
Missing	3	1.2
Age at Enrollment
18–21	55	22.3
22–26	45	18.2
27–31	41	16.6
32–36	40	16.2
37–40	66	26.7
Age at DX
0–5	20	8.1
6–11	23	9.3
12–17	68	27.5
18–23	37	15
24–30	49	19.8
31+	50	20.2
Cancer-Related Treatment
Surgery	140	56.7
Radiation	152	61.5
Chemotherapy	214	86.6
Bone marrow/stem cell transplant	50	20.2
Years Since Treatment
2–4	99	40.1
5–9	91	36.8
10–15	27	10.9
15+	28	11.3
missing	2	0.8
Cancer Diagnosis
Hodgkin’s Lymphoma	50	20.2
Leukemia	49	19.8
Brain tumor	30	12.1
Non-Hodgkin’s Lymphoma	30	12.1
Testicular	27	10.9
Breast	24	9.7
Sarcomas	20	8.1
Other	17	6.9
Distress Thermometer	N	%
0	49	19.8
1	37	15
2	39	15.8
3	33	13.4
4	15	6.1
5	20	8.1
6	20	8.1
7	19	7.7
8	13	5.3
9	0	0
10	2	0.8
SCID Criteria	N	%
SCID Diagnosis	44	17.8
Significant SCID Symptoms no Diagnosis	20	8.1
No Significant SCID Symptoms	183	74.1

Forty-four survivors (17.8%) were classified with a SCID diagnosis, 20 (8.1%) were classified with Significant SCID symptoms, and 183 (74.1%) were classified as having No Significant SCID Symptoms (Table 1; Frequency of specific diagnoses shown in Supplemental Material 2). On the DT, 124 survivors (50.2%) endorsed low levels of distress (DT <3), 88 (35.6%) endorsed moderate distress (scores 3–6), and 34 (13.8%) endorsed high distress levels (DT ≥ 7). The relationship between participants’ responses on the DT and the SCID is presented in Figure 1.

Figure 1. Sample Distribution on the DT and SCID.

The relationship between participants’ responses on the DT and the SCID is presented in this figure; participants were classified into one of three categories on the SCID, based on diagnostic outcome, and self-reported a distress level on the DT.

Classification Agreement between the DT and SCID Diagnoses

In ROC analyses, the DT demonstrated moderate discrimination between participants with and without a SCID diagnosis (Figure 2; AUC=.803). Evaluating sensitivity and specificity of each potential DT cut-off score (Table 2), it was found the cut-point of ≥5 had limited sensitivity (68.18%) to detect survivors with SCID diagnoses; though with specificity of 78.33%, it would correctly identify a reasonable proportion of those without a SCID diagnosis.

Figure 2. ROC Curve Depicting Relationship of DT Scores to A) SCID Diagnosis and B) SCID Diagnosis or Significant Symptoms.

In ROC analyses, the DT demonstrated moderate discrimination between participants with and without a SCID diagnosis.

Table 2.

Sensitivity and specificity of the DT for Detecting Survivors with A) Any SCID Diagnoses and B) Any SCID Diagnosis or Any Significant Symptoms

DT Cut-off Score	A. SCID Diagnosis		B. Diagnosis or Significant Symptoms
	Sensitivity	Specificity	Sensitivity	Specificity
≥0	100.00	0.00	100.00	0
≥1	97.73	23.65	98.44	26.23
≥2	90.91	40.39	92.19	44.26
≥3	84.09	58.13	85.94	63.39
≥4	81.82	73.89	78.13	78.69
≥5	68.18	78.33	67.19	83.06
≥6	54.55	85.22	53.13	89.07
≥7	38.64	91.63	34.38	93.44
≥8	20.45	97.04	17.19	97.81
≥9	4.55	100.00	3.13	100.00
=10	4.55	100.00	3.13	100.00

To demonstrate implications of these results for making screening decisions, we applied this sensitivity and specificity to a hypothetical example of screening 100 survivors (20% with a psychiatric diagnosis), with the DT (Figure 3). In this example, we would expect 14 (68%) of the 20 survivors with a diagnosis to be “True Positives,” with a DT score ≥5 leading them to be correctly identified as having a psychiatric diagnosis. The other 6 survivors with a psychiatric condition would be “false negatives” on the DT (DT <5) meaning 31.82% of survivors with a SCID diagnosis would be sent home without further evaluation. For the 80 survivors with no psychiatric condition, 62 (78%) with DT scores <5 would be “True Negatives,” but the 18 with DT scores ≥5 would be erroneously sent for further evaluation (False Positives). In all, 32 of the 100 survivors would be referred for mental health services, though less than half (14 or 43.75%) would actually have a diagnosable condition.

Figure 3. Expected Clinical Decisions When Using the DT with Cut-off score ≥ 5 to Screen YA Cancer Survivors.

This figure demonstrates the implications of our results for making screening decisions, with applied sensitivity and specificity to a hypothetical example of screening 100 survivors (20% with a psychiatric diagnosis) with the DT.

As expected, lower cut-off scores increase sensitivity of the DT, or the detection of survivors with a SCID diagnosis, but decrease specificity indicating that many more false positives would result. For example, the cut-off score ≥2 would correctly identify 40 of 44 survivors with a SCID diagnosis (90.9%), but would falsely classify 121 (59.6%) of 203 survivors with no SCID diagnosis as being distressed. No DT cut-off score met study criteria for sensitivity (>.85) and specificity (>.75) for detecting SCID diagnoses; the cut-off score ≥4 came closest, but still failed to identify 18.18% of survivors with a SCID diagnosis.

When agreement between the DT and the SCID was evaluated separately for depressive and anxiety disorders, results were similar. In ROC analyses (Supplemental Material 3), the DT was somewhat better in detecting depressive than anxiety disorders (AUC = .831 v .773). The ≥4 DT cut-off had good sensitivity (87.50%) and marginal specificity (71.63%) for detecting survivors with depressive disorders, and had somewhat less robust performance for anxiety disorders (sensitivity = 78.79%; specificity = 70.56%; Table 3). No potential DT cut-off scores met study criteria for detecting depressive or anxiety disorders on the SCID.

Table 3.

Sensitivity and Specificity of the DT for Detecting Survivors with A) SCID Diagnoses and B) SCID Diagnosis or Significant Symptoms Separate for Anxiety and Depression

A. Diagnoses					B. Diagnoses or Significant Symptoms
DT Cut- off Score	DEPRESSIVE		ANXIETY		DEPRESSIVE		ANXIETY
	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Sensitivity	Sensitivity	Specificity
≥0	100.00	0.00	100.00	0.00	100.00	0	100.00	0
≥1	100.00	22.79	96.97	22.43	97.67	23.53	98.39	25.95
≥2	96.88	39.53	87.88	38.32	97.67	41.67	91.94	43.78
≥3	90.63	56.74	78.79	55.14	93.02	59.80	85.48	62.70
≥4	87.50	71.63	78.79	70.56	81.40	73.53	77.42	77.84
≥5	71.88	76.28	69.70	76.17	69.77	78.43	66.13	82.16
≥6	56.25	83.26	54.55	83.18	58.14	85.78	51.61	88.11
≥7	40.63	90.23	39.39	90.19	39.53	91.67	35.48	93.51
≥8	28.13	97.21	24.24	96.73	23.26	97.55	17.74	97.84
≥9	6.25	100.00	6.06	100.00	4.65	100.00	3.23	100.00
=10	6.25	100.00	6.06	100.00	4.65	100.00	3.23	100.00

Classification Agreement between the DT and Significant SCID Symptoms

Because of the importance of identifying cancer survivors with symptoms of psychological distress even if they do not meet criteria for a psychiatric diagnosis,^21,37 we evaluated the DT’s ability to identify survivors classified as having either significant SCID symptoms or a SCID diagnosis. In ROC analyses, the DT demonstrated overall good discrimination between these two groups (Figure 2; AUC=.829). A cut-off score of ≥5 had limited sensitivity to detect survivors with significant SCID symptoms or diagnosis (67.19%), with almost 1 of every 3 survivors meeting this SCID criterion missed on the DT (Table 2). The ≥4 cut-off score had better, but still limited sensitivity of 78.13%. A DT cut-off score of ≥3 met study criteria with sensitivity of 85.94%, but had poor specificity (63.39%). Results were similar when these analyses were repeated separately for depression and anxiety. In ROC analyses (Supplemental Material 3), the DT performed similarly in detecting depressive (AUC = .837) and anxiety symptoms (AUC = .822). A DT cut-off of ≥ 3 was necessary to achieve acceptable levels of sensitivity for either depressive (93.02%) or anxiety symptoms (85.48%), and these were associated with unacceptably low specificity (< .65; Table 2). No potential DT cut-off score met study criteria for detecting individuals with significant SCID symptoms either as a group, or separately for those with significant depressive or anxiety symptoms.

Discussion

To examine the validity of the DT for screening YA cancer survivors, we compared it to a “gold standard” diagnostic measure for psychiatric disorders and found that recommended DT cut-off scores were not effective at identifying survivors with a psychiatric diagnosis. Applying a DT cut-off score of 5 missed almost a third of survivors with a psychiatric diagnosis. An alternative cut-off of 4 performed better, but still missed more than 1 in 5 survivors meeting psychiatric diagnostic criteria. In fact, no alternative DT cut-of score met study criteria for acceptable sensitivity and specificity. This insensitivity to psychiatric disorders is disappointing in a screening instrument, since the central rationale for a screening is to bring to light otherwise occult psychological disturbances.³⁸ Diagnoses may not be the only aspect of distress we should be concerned about identifying in cancer survivors, but if the DT misses these more serious mental health disturbances, it fails to identify the most distressed survivors that are likely to be seen by health care providers. Moreover, when we evaluated the DT’s potential to identify YA survivors with either a SCID diagnosis or significant psychological symptoms, results were similar.

Though prior studies have supported the DT as a rapid screening measure for oncology patients, our findings that it does not reliably identify psychological problems in cancer survivors are not new. Studies from our center comparing the DT to longer self-report ratings scales found the DT was not an accurate measure of psychological symptoms in survivors of childhood²⁷ or adult onset cancers,²⁵ and similar findings have been reported by other investigators.²⁴ Those prior studies, however, only compared the DT to other self-report rating scales which can have their own limitations,^20,23,25 making it impossible to conclude with certainty that lack of agreement between the measures was a shortcoming of the DT. By comparing the DT directly to a structured diagnostic interview, our results here eliminate this concern and demonstrate the limitations of the DT compared to this gold standard measure.

In evaluating the DT, we applied criteria adapted from prior studies^25,27 specifying that a screening measure should have sensitivity ≥ 85% and specificity ≥ 75% to be acceptable for clinical use. These criteria are not arbitrary, but based on high levels of precision generally expected in medical screening,³⁸ prior research showing that many psychological self-report measures meet these criteria,^34–36 as well as concern about high rates of misidentification which can have a negative impact on individual patients and systems of care. In our example of the clinical decisions resulting from applying the DT to YA survivors, we noted that almost 30% of YA survivors with a psychiatric diagnosis would be missed using the standard cut-off score. These false negative decisions not only fail to identify survivors with mental health needs, but may actually be an additional obstacle to obtaining needed care as providers and patients have been misled into thinking it is not indicated. At the same time, even with acceptable specificity (78%) in the example, it is notable that less than half of the YA survivors recommended for additional assessment would actually have a psychiatric disorder. While the DT would have been evaluated more favorably had we applied more relaxed criteria, this would ignore implications for screening decisions and potentially overstate its utility as a clinical tool in this population. However, data reported here on the screening properties of the DT at each potential cut-off score are useful for users wanting to evaluate the DT for different selection purposes.

It should be noted that the study sample was limited to cancer survivors ages 18–40, so results cannot be generalized to other age groups. In addition, a non-random sample of survivors from a single site may over- or underrepresent the level of psychological problems in the larger YA population. However, this is not likely to be a serious threat to the validity of our study since sensitivity and specificity of a screening measure are not a function of the proportion of individuals with the condition of interest.³⁸ Finally, though the DT and SCID were administered on the same day, each measure inquires about symptoms during different reference periods—one week and one month respectively. It is common practice in this type of study³⁶ to use instruments with their standard reference periods, and though it is unlikely that psychiatric disorders would fully resolve over a short period of two to three weeks, symptom fluctuations over the course of a month could contribute to lack of agreement between the two measures. Future studies could evaluate this by inquiring in more detail about symptom timing and duration on all measures.

Despite these limitations, the results have important implications for identifying emotional health needs in YA cancer survivors. Our findings confirm prior reports that applying the DT with conventional cut-off scores to YA survivors fails to detect up to 30% of those with significant psychological distress. Moreover, no potential cut-off score met criteria for sensitivity and specificity, indicating that the DT alone should not be used for clinical decision making in this population. Though “ultra-brief” screening measures like the DT may be appealing because of their brevity, the limited information they provide may also limit their accuracy.^13,24 Ultimately, longer instruments with more items may be more accurate in identifying serious symptoms of anxiety and depression, and studies of their validity in YA survivors should be encouraged. Alternatively, using the DT with a low cut-off score to achieve high sensitivity, along with a second screening measure that can eliminate false positives on the DT may also be an important strategy to pursue. For example, adding brief measures of distress duration and associated level of impairment as Mitchell et al. have suggested,²⁶ may be particularly useful as these are critical to defining psychiatric diagnoses, but are not assessed by the DT or most self-report symptoms measures.