Screening Young Adult Cancer Survivors with the PROMIS Depression Short-Form (PROMIS-D-SF): Comparison with a Structured Clinical Diagnostic Interview

Abstract

Objective

To evaluate the PROMIS Depression short-form (PROMIS-D-SF) as a screening measure for young adult cancer survivors (YACS), compared to a structured diagnostic interview.

Methods

249 YACS (aged 18–40) completed the PROMIS-D-SF and Structured Clinical Interview for the DSM-IV (SCID). Based on the SCID, participants were classified as having: 1) SCID depression diagnosis; 2) Depression symptoms without SCID diagnosis; or 3)No depression symptoms; ROC analyses evaluated PROMIS-D-SF and SCID concordance and sensitivity and specificity of PROMIS-D-SF cut-off scores.

Results

The PROMIS-D-SF had overall good agreement with the SCID on both depression diagnosis (AUC = .89) and presence of depressive symptoms (AUC = .83). A PROMIS-D-SF cut-off of t ≥ 53.2 came closest to meeting study criteria for detecting a SCID depression diagnosis (sensitivity ≥ .85; specificity ≥ .75), with sensitivity .81 and specificity .74. For identifying survivors with depression symptoms, a t-score cut-off of 49.4 had slightly superior sensitivity (.84) and inferior specificity (.64). In hypothetical screening examples, these cut-offs led to moderate levels of missed cases (15–19%) and a high proportion of clinical referrals that were unnecessary by SCID criteria (56–70%).

Conclusions

The PROMIS-D-SF demonstrates moderately strong concordance with depressive diagnoses and symptoms measured by the SCID, but cut-off scores did not meet study criteria for clinical screening. The PROMIS-D-SF may be a useful for assessing depression in YACS, but limitations in sensitivity and specificity identified in this study are likely to limit its utility as a stand-alone screening instrument in this population.

Introduction

While the majority of young adult cancer survivors (YACS) are well-adjusted, several studies indicate they are more prone to adjustment problems than older survivors,^1–3 and have higher rates of depressive symptoms compared to same-age peers.^4–8 Follow-up care guidelines recommend psychological screening for all cancer survivors and YACS in particular, but offer limited practical guidance about how screening should be conducted or what measures are appropriate. While a variety of self-report symptom checklist measures have been advocated for this purpose,^9–11 few studies have specifically examined the validity of self-report depression screening measures for YACS, and even fewer have compared these measures to diagnostic interviews standard in psychiatric assessment research.^12–14 This is highly problematic because without comparisons to “gold standard” diagnostic interviews, it is difficult to determine whether these measures accurately identify patients with clinically significant symptoms and diagnoses.

The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed to measure patient-reported outcomes in individuals with a wide range of diseases and demographic characteristics.^15–17 By evaluating a large number of items drawn from existing self-report measures in both healthy and clinical groups, PROMIS aimed to develop optimized item pools and norm-based scoring. Making use of item response theory, PROMIS features on-line computer-adaptive testing (CAT) that individually tailors test administration by selecting items based on a respondent’s previous responses. While CAT allows for precise measurement with limited burden, it does require that measures be administered on the web-based PROMIS Assessment Center. For settings where computerized testing is not feasible, static PROMIS short-form measures have been developed, including the PROMIS Depression Short-Form (PROMIS-D-SF).^{16, 18}

Though studies support reliability and sensitivity to change of the PROMIS-D-SF, research on its accuracy as a screening measure has been limited. The PROMIS-D-SF was introduced without a recommended case-rule, and to date specific clinical cut-off scores have not been consistently supported.^{15, 18–21} Consequently studies comparing the PROMIS-D-SF to gold-standard psychiatric diagnostic measures are critical to supporting its clinical use.^{16, 19} As PROMIS measures are proposed for screening cancer survivors, including YACS, ^{22, 23} we set out to validate the PROMIS-D-SF against a structured diagnostic interview in order to determine its accuracy and the most appropriate cut-off score for identifying clinically significant depression in this group.

Method

Participants

As previously described,^{12, 14} patients at our cancer center were recruited to participate in E-Quest, a study of self-report anxiety and depression measures if they were age 18–40, English-speaking, ≥ three years from cancer diagnosis, and ≥ two years since treatment completion. Consenting participants completed self-report measures including the PROMIS-D-SF, before completing the Structured Clinical Interview for the DSM-IV (SCID), an interviewer administered diagnostic interview. Study procedures were approved by the cancer center’s institutional review board and were completed on a single day.

Measures

Patient-Reported Outcomes Measurement Information System Depression Short-Form (PROMIS-D-SF)^{15, 24}

The PROMIS-D-SF (version 8a) is an 8-item self-report measure asking respondents to describe their depression in the prior 7 days by rating items on a 5-point scale. The 8 PROMIS-D-SF items were selected from the PROMIS Depression item bank to most closely approximate depression scores obtained from CAT administration using the full 28-item bank. Participants completed the PROMIS-D-SF on paper; following scoring guidelines,²⁴ raw scores were converted to t-scores (mean = 50, SD = 10).

Structured Clinical Interview for the DSM-IV (SCID)²⁵

The SCID is a widely used semi-structured clinical interview for psychiatric diagnoses based on DSM-IV criteria.^26,27 Items closely follow diagnostic decision trees and skip patterns ensure respondents without minimal criteria for a diagnosis are not administered additional items. Interviews were conducted by a single interviewer, with a Master’s degree in psychology (JEB) and trained in SCID administration and scoring who was blind to participants’ PROMIS-D-SF responses.

Using the research version of the SCID, which permits selection of relevant diagnostic modules, the Major Depressive Disorder, Minor Depressive Disorder, Dysthymia, Mixed Anxiety and Depressive Disorder, and Adjustment Disorder modules were administered. Initial questions from these modules (Table 1) were administered as part of the SCID screening module and were used as the basis for classifying individuals as having depressive symptoms (described below).

Table 1:

Depression Symptoms on the SCID Screener

Symptom Description	SCID Item #	DSM-IV-TR Criteria
Depressive Symptoms
1. Depressed mood most of the time for 2 weeks in the past month	A1	Major/Minor Depressive Episode Criterion A
2. Diminished interest most of the time for 2 weeks in the past month	A2	Major/Minor Depressive Episode Criterion A
3. Depressed mood most of the time for past 2 years	A163	Dysthymic Disorder Criterion A
4. Persistent or recurrent dysphoric mood state lasting ≥ 1 month	J20	Mixed Anxiety-Depressive Disorder Criterion A

Classification of Participants on the SCID

SCID interviews were scored using standard diagnostic algorithms.²⁵ Only current symptoms and diagnoses (present within the prior 30 days) were scored. Based on SCID responses, participants were classified into one of three categories; 1) Participants with SCID Depression Diagnosis (participants meeting criteria for any SCID depression diagnosis); 2) Participants with Depression Symptoms without diagnosis, (i.e. endorsed ≥ 1 depression symptom on the SCID screener (Table 2), but not meeting SCID depression diagnosis criteria); and 3) Participants with No Depression Symptoms (i.e., endorsed no depression symptoms on the SCID screener). Following standard SCID scoring, 5 participants were diagnosed with sub-threshold SCID depression diagnoses because the interview indicated diagnostic criteria were likely met, but the participant could not definitively recall some relevant detail; these participants were included with the participants having confirmed diagnoses in all analyses.

Table 2:

Sample Description (N= 249)

Gender	N	%
Male	125	50.2
Female	124	49.8
Age at Enrollment
18–21	54	21.7
22–26	45	18.1
27–31	42	16.9
32–36	41	16.4
37–40	67	26.9
Age at Cancer Diagnosis
<21	124	49.8
≥21	125	50.2
Years after Cancer Treatment Completion
2–4	101	40.6
5–9	91	36.6
10–15	27	10.8
15+	28	11.2
Missing	2	0.8
Cancer Diagnosis
Hodgkin’s Lymphoma	50	20.1
Leukemia	50	20.1
Brain tumor	30	12.1
Non-Hodgkin’s Lymphoma	28	11.2
Testicular	26	10.5
Breast	24	9.6
Sarcomas	20	8.0
Other	21	8.4
SCID Criteria	N	%
Depressive Diagnosis	32	12.9
Dysthymic Disorder	9
Major Depression	7
Mixed Anxiety & Depressive Disorder	6
Minor Depressive Disorder	5
Adjustment Reaction with depressive features	5
Depressive Symptoms, no Diagnosis	29	11.6
No Depressive Symptoms	188	75.5

Statistical Analysis

Receiving Operator Characteristic (ROC) curves were used to determine concordance of the PROMIS-D-SF scale with SCID depression diagnoses. As there may be interest in identifying YACS with depressive symptoms without regard to diagnosis, these analyses were repeated to evaluate concordance with depressive symptoms by combining YACS with a depression diagnosis, and YACS with depression symptoms without a diagnosis into a single group. Area under the curve (AUC) quantified discrimination across the range of possible scores, with AUC values ≥ 0.80 reflecting good discrimination and values ≥ 0.90 reflecting excellent discrimination.²⁸ In addition, we reported three conditional probabilities for each cut-off score evaluated: sensitivity, or true positive rate, reflecting a screening test’s likelihood of providing a correct positive result for individuals who have the condition of interest; specificity, or true negative rate, reflecting likelihood a screening test provides correct negative result to individuals who do not have the condition, and Total Percent Correct (aka, total predictive value or accuracy) reflecting proportion of correct screening results.^{29, 30} Clinical screening programs generally select criteria that prioritize high sensitivity^{31, 32} in order to ensure that most, if not all patients with the condition are identified by the screening instrument. We determined a priori that a PROMIS-D-SF cut-off score with sensitivity ≥ 85% and specificity ≥ 75% would be adequate to be recommended for screening purposes using criteria based on clinical considerations and studies demonstrating depression screening instruments can demonstrate high sensitivity and maintain robust specificity.^{18, 19, 31, 33}

Results

Distribution of the Sample on the SCID and PROMIS-D-SF

During the study period, 349 eligible survivors were approached, 250 (71.6%) enrolled, and 249 (125 males & 124 females; mean age =29.5 yrs, SD=7.32) completed the PROMIS-D-SF and SCID (Table 2). Based on the SCID, 32 of these 249 survivors (12.9%) met criteria for a SCID Depression Diagnosis, 29 (11.6%) had Depression Symptoms without a diagnosis, and 188 (75.5%) were classified as having No Depression Symptoms. Mean PROMIS-D-SF t-score was 48.85 (SD = 8.62); 141 survivors (56.6%) scored at or below the population mean (t-score ≤ 50.0), 78 (31.3%) scored within 1 SD above the population mean (t-scores  = 50.0–59.9), and 30 (12.0%) scored ≥ 1 SD above the population mean (t-score ≥ 60). The relationship between PROMIS-D-SF scores and the SCID is presented in Supplemental Figure 1.

The PROMIS-D-SF and SCID Depression Diagnoses

The ability of the PROMIS-D-SF to identify patients with a depression diagnosis was evaluated using ROC analyses. AUC was 0.879 (Supplemental Figure 2), indicating the PROMIS-D-SF had good overall discrimination compared to the SCID. Evaluating specific PROMIS-D-SF cut-off scores (Table 3), it was found a t-score cut-off of ≥ 63 was most accurate for distinguishing between YACS with a SCID depression diagnosis and those without a depression diagnosis, with >90% of all subjects correctly classified. This cut-off could be useful in applications where overall classification is most important, but its low sensitivity (41%) indicates it would miss almost 60% of YACS with depression diagnoses, making it inappropriate for clinical screening. A cut-off score of ≥ 53.2 came closest to meeting our criteria for screening, with a sensitivity of 81% and specificity of 74%. As expected, lower cut-off scores had improved sensitivity but decreased specificity; a cut-off of ≥ 50.9 would detect 88% of YACS a with SCID depression diagnosis, but with specificity of only 63%, it would tend to “over-diagnose” depression in survivors without a true diagnosis.

Table 3:

Sensitivity and Specificity of the PROMIS-D-SF for Detecting Depression Diagnoses and Significant Depression Symptoms on the SCID

PROMIS DEP t-Scores	Depression Diagnosis (N=32/249)			Depression Symptoms (N=61/249)
Alternative Cut-Offs	Sensitivity	Specificity	% Correct	Sensitivity	Specificity	% Correct
47.5	1 (0.87–1)	0.46 (0.39–0.53)	53.0	0.90 (0.79–0.96)	0.50 (0.43–0.57)	59.8
49.4	0.91 (0.74–0.98)	0.59 (0.52–0.66)	63.1	0.84 (0.71–0.91)	0.64 (0.57–0.71)	69.1
50.9	0.88 (0.70–0.96)	0.63 (0.56–0.69)	66.3	0.82 (0.70–0.90)	0.69 (0.62–0.76)	72.3
52.1	0.81 (0.63–0.92)	0.68 (0.61–0.74)	69.9	0.70 (0.57–0.81)	0.72 (0.65–0.78)	71.9
53.2	0.81 (0.63–0.92)	0.74 (0.68–0.80)	75.1	0.69 (0.56–0.80)	0.79 (0.72–0.84)	76.3
54.1	0.75 (0.56–0.88)	0.78 (0.72–0.84)	77.9	0.64 (0.51–0.76)	0.83 (0.77–0.88)	78.3
55.1	0.75 (0.56–0.88)	0.82 (0.76–0.87)	81.1	0.56 (0.43–0.68)	0.85 (0.78–0.89)	77.5
55.9	0.69 (0.50–0.83)	0.84 (0.79–0.89)	82.3	0.52 (0.39–0.65)	0.87 (0.81–0.91)	78.7
56.8	0.69 (0.50–0.83)	0.87 (0.82–0.91)	84.7	0.51 (0.38–0.64)	0.90 (0.84–0.94)	80.3
57.7	0.66 (0.47–0.81)	0.90 (0.85–0.93)	86.7	0.48 (0.35–0.61)	0.93 (0.88–0.96)	81.5
58.5	0.66 (0.47–0.81)	0.91 (0.86–0.95)	88.0	0.46 (0.33–0.59)	0.94 (0.89–0.97)	81.9
59.4	0.63 (0.44–0.78)	0.94 (0.90–0.97)	90.0	0.41 (0.29–0.54)	0.96 (0.91–0.98)	82.3
60.3	0.59 (0.41–0.76)	0.95 (0.91–0.97)	90.4	0.39 (0.27–0.53)	0.97 (0.93–0.99)	82.7
61.2	0.53 (0.35–0.70)	0.96 (0.92–0.98)	90.4	0.33 (0.22–0.46)	0.97 (0.93–0.99)	81.1
62.1	0.47 (0.30–0.65)	0.96 (0.93–0.98)	90.0	0.30 (0.19–0.43)	0.97 (0.94–0.99)	80.7
63	0.41 (0.24–0.59)	0.98 (0.95–0.99)	90.8	0.25 (0.15–0.38)	0.99 (0.96–1)	80.7
63.9	0.38 (0.22–0.56)	0.98 (0.95–0.99)	90.4	0.23 (0.14–0.36)	0.99 (0.96–1)	80.3
64.9	0.34 (0.19–0.53)	0.99 (0.96–1)	90.4	0.21 (0.12–0.34)	0.99 (0.97–1)	80.3
65.8	0.22 (0.10–0.40)	0.99 (0.96–1)	89.2	0.13 (0.06–0.25)	0.99 (0.97–1)	78.3
66.8	0.19 (0.08–0.37)	1 (0.97–1)	89.2	0.10 (0.04–0.21)	0.99 (0.97–1)	77.5
67.7	0.16 (0.06–0.34)	1 (0.98–1)	89.2	0.08 (0.03–0.19)	1 (0.98–1)	77.5
68.7	0.13 (0.04–0.30)	1 (0.98–1)	88.8	0.07 (0.02–0.17)	1 (0.98–1)	77.1
69.7	0.13 (0.04–0.30)	1 (0.98–1)	88.8	0.07 (0.02–0.17)	1 (0.98–1)	77.1
70.7	0.03 (0–0.18)	1 (0.98–1)	87.6	0.02 (0–0.10)	1 (0.98–1)	75.9
71.7	0 (0–0.13)	1 (0.98–1)	87.1	0 (0–0.07)	1 (0.98–1)	75.5
72.8	0 (0–0.13)	1 (0.98–1)	87.1	0 (0–0.07)	1 (0.98–1)	75.5
73.9	0 (0–0.13)	1 (0.98–1)	87.1	0 (0–0.07)	1 (0.98–1)	75.5
75	0 (0–0.13)	1 (0.98–1)	87.1	0 (0–0.07)	1 (0.98–1)	75.5
76.4	0 (0–0.13)	1 (0.98–1)	87.1	0 (0–0.07)	1 (0.98–1)	75.5

To demonstrate implications of these results for screening, we examined how the PROMIS-D-SF ≥ 53.2 t-score cut-off would operate in a hypothetical sample of 100 survivors with 13% having a depression diagnosis (similar to the 12.9% in our sample; Figure 1). In this example, we would expect the PROMIS-D-SF to correctly identify 11 of 13 survivors with a depressive diagnosis making them “True Positives”; the other 2 of 13 would be “False Negatives,” individuals with a SCID depression diagnosis but PROMIS-D-SF scores below the cut-off who would not be referred for further evaluation. Of 87 survivors with no depression diagnosis, 64 (74%) would be expected to be correctly classified as “True Negatives,” and the other 23 would be “False Positives.” Of the 34 survivors referred for services in this example (11 True Positives & 23 False Positives), less than a third (32%) would actually have a SCID depressive diagnosis, highlighting the potential burden that screening with even moderate specificity can have for a health care system, especially when the prevalence of a disorder is low.

Figure 1:

Expected Clinical Decisions Using PROMIS-D-SF to Screen for Depression Diagnosis (Cut-off score ≥ 53.2)

The PROMIS-D-SF and SCID Depression Symptoms

Utility of the PROMIS-D SF for identifying YACS with ≥1 depression symptom, with or without a diagnosis, was also evaluated using ROC analysis. AUC of 0.823 (Supplementary Figure 2), indicated the PROMIS-D-SF had good overall discrimination compared to the SCID. A cut-off t-score of ≥ 60.3 was overall most accurate for distinguishing YACS with depression symptoms from those with no depression symptoms, with 82.7% of individuals correctly classified. However, the low sensitivity (39%), indicating more than 60% of YACS with depressive symptoms would be missed, underscores its lack of suitability for clinical screening. A cut-off of ≥ 49.4 came closest to meeting study criteria with sensitivity of 84% and specificity of 64%. A lower cut-off score of ≥ 47.5 had improved sensitivity of 90%, but a low specificity of 50%, that would lead to half of YACS with no symptoms being incorrectly classified.

Screening implications of the PROMIS-D-SF ≥ 49.4 t-score cut-off were examined in hypothetical sample of 100 survivors similar to our sample (Figure 2). Of 25 survivors with depressive symptoms, we would expect 21 “True Positives” and 4 “False Negatives.” Of 75 survivors without depression symptoms, we would expect 52 “True Negatives,” and 23 “False Positives.” Of 44 survivors referred for services (21 True Positives & 23 False Positives), less than half (48%) would have depressive symptoms on the SCID.

Figure 2:

Expected Clinical Decisions Using PROMIS-D-SF to Screen for Depression Symptoms (Cut-off score ≥ 49.4)

Discussion

While studies have previously evaluated the PROMIS-D-SF as a screening instrument in cancer and other medical populations,^{18, 19} we are not aware of any study of YACS to do so. By examining concordance of the PROMIS-D-SF with a structured diagnostic interview, our results provide novel information to guide its application in this population at risk for depression. Consistent with previous studies,^{18, 19} we found overall good concordance between the PROMIS-D-SF and the SCID, but when we evaluated the PROMIS-D-SF as a screening instrument for individual YACS, results were not robust. All PROMIS-D-SF cut-off scores fell short of our minimum criteria for both high sensitivity (≥ 85%) and moderate specificity (≥ 75%), and results indicate screening YACS using the PROMIS-D-SF would lead to both a significant proportion of symptomatic survivors being missed, and a significant proportion of clinical referrals being unnecessary according to SCID criteria.

While it is unfortunate the PROMIS-D-SF demonstrated these limitations as a screening tool for YACS, it is not altogether unexpected. The PROMIS Depression scale was developed to measure depressive symptoms along a continuum, and no specific efforts were made to include items reflecting the depressive symptoms seen in psychiatric disorders.¹⁵ Moreover, prior reports have found the PROMIS-D-SF has only moderate agreement with structured diagnostic interviews in patients with medical conditions including cancer.^{18, 19}

Results should be considered in light of study limitations. A moderate size sample drawn from a single center may not represent prevalence of depression in YACS at large, especially as we were unable to characterize health outcomes of non-participants. However, as sensitivity and specificity are not affected by prevalence of the condition under study,^{29, 30} this is unlikely to bias our results. We applied diagnostic criteria from DSM-IV, not the recently introduced DSM-5. As depression criteria changed very little in DSM-5,³⁴ we believe results using DSM-5 would be highly similar, but this can be confirmed only with future studies. Of note, the PROMIS-D-SF and SCID were administered on the same day, using their standard reference periods—one week and one month, respectively. While psychiatric disorders are unlikely to completely resolve in short periods, it is possible symptom changes over the course of a month could limit agreement between the PROMIS-D-SF and SCID. Investigating temporal changes in symptoms should be a priority for future studies. Finally, we specified that screening measures should demonstrate sensitivity ≥ 85% and specificity ≥ 75% for clinical use. These criteria are consistent with those previously applied in psychological screening studies in oncology^{12, 18, 19, 35, 36} and precision typically demonstrated by medical testing.^{29, 37, 38} We report sensitivity and specificity for a range of PROMIS-D-SF cut-off scores for readers who may prefer to apply different standards for evaluating the PROMIS-D-SF, and explore how they may operate in samples with varying prevalence of depression symptoms and diagnosis.

Despite these limitations, the findings have important implications for using the PROMIS-D-SF in YACS. In medical screening, the decision to refer a patient for further evaluation is made principally, if not solely, on results of a test that detects signs of illness that cannot be identified in routine clinical care.²⁹ For example, a blood test showing a high hemoglobin A1c, indicating effects of high blood sugar, can be used to refer otherwise asymptomatic patients for further evaluation for diabetes. A highly accurate depression screen in YACS would be very useful for oncologists and other providers who could rely on it in a similar manner to determine which YACS require further evaluation for depression. Unfortunately, consistent with studies of several other symptom checklist screening measures,^{12, 14, 18, 19, 35, 36, 39} our results indicate the PROMIS-D-SF is not sufficiently sensitive and specific to be used as a stand-alone screening tool for depression in YACS.

Practice guidelines^{9–11, 40} increasingly call for psychological screening to determine when cancer patients require mental health care, yet definitive empirical evidence supporting specific screening measures is very limited. For many investigators, the way forward is to continue measure development and validation until psychological screening measures for cancer survivors are supported by strong empirical evidence. Others strongly question the continued application of the medical diagnostic model to mental health screening in oncology. Salmon et al.⁴¹ for example, argue that diagnostic models and psychometric analyses fail to consider the “fundamental questions facing the field” of psychological screening in oncology. According to their assessment, facing these questions is likely to “complicate our understanding of screening rather than simplify it; … [and] show that deciding whether a patient has psychological needs and how these should be met, is too complex to be reduced to a simple screen for distress.”⁴¹ Although we believe it may be premature to end the search for psychometrically robust psychological screening measures in oncology, we appreciate the importance of recognizing the limitations of current measures, while also considering how they may be responsibly applied to clinical care.

No screening measure is perfect, and screening benefits will depend largely on availability and feasibility of appropriate follow-up. In the case of the PROMIS-D-SF, it is important to consider how an imperfect measure may be valuable to providers evaluating depression in YACS. Medical practitioners routinely weigh and integrate different types of information including history, vital signs, laboratory values, along with patients’ signs and symptoms to make clinical formulations and plan follow-up. With adequate information, including performance of specific cut-off scores, providers could similarly integrate the PROMIS-D-SF into their clinical assessment. For example, though low scorers on the PROMIS-D-SF are unlikely to have significant depression, clinicians aware of moderate sensitivity may consider asking these YACS more about mood symptoms to avoid missed depression cases, especially in the context of other risk factors (e.g., family history, poor health status). Similarly, recognizing the likelihood of false positives, providers could choose to inquire about symptom intensity, associated impairment and duration, to better discern when high PROMIS-D-SF scores should lead to a mental health referral for YACS and when they should not. While we advocate for continued psychometric research that may someday solve the challenge of psychological screening, given current limitations of the PROMIS-D-SF and other symptom checklist measures,^{18, 19, 35, 36, 39} using them to inform mental health assessment in oncology in this way is likely to better serve the needs of YACS and health care systems than implementing them as stand-alone screens.