Translating Cognitive Vulnerability Theory Into Improved Adolescent Depression Screening: A Receiver Operating Characteristic Approach

Abstract

Traditionally, screening research tests how well a given symptom inventory can identify a concurrent depressive episode. Although developmental psychopathology could inform screening protocols for a myriad of depression outcomes (e.g., prospective depressive episodes), approaches typically used in research make it difficult to translate these findings. Using a translational analytic approach and multiwave longitudinal study design, we examined how screening for cognitive vulnerabilities (rumination, dysfunctional attitudes, and attributional style) may improve our ability to identify concurrent depressive episodes, prospective depressive episodes, first lifetime episodes of depression, and recurrent major depressive episodes. There were 473 sixth-grade (early adolescents) and ninth-grade (middle adolescents; Age_M = 13.15, Age_SD = 1.62) students who completed baseline self-report cognitive vulnerability and depressive symptom measures. At baseline and every 6 months for 3 years, pediatric depression interviews were completed by the caregiver and youth. A receiver operating characteristic (ROC) approach was utilized to test our aims. Distinct algorithms best forecasted our depression outcomes. Rumination and attributional style emerged as unique and incrementally valid predictors for prospective episodes after controlling for baseline depressive symptoms. Rumination was the only unique predictor for first lifetime depressive episodes. For recurrent major depression, rumination in early adolescence and attributional style in middle adolescence served as incremental predictors beyond baseline depressive symptoms. Proposed cutoffs and diagnostic likelihood ratios are offered for algorithms for each depression outcome. Assessing cognitive vulnerability represents a feasible method to improve depression screening initiatives. Using an ROC-informed approach can help prevention initiatives better leverage the considerable gains made within developmental psychopathology research.

Developmental psychopathology aims to understand the multifaceted processes that contribute to the onset and maintenance of psychological distress (Cicchetti & Toth, 2009). The discipline plays a key role in the formation of evidence-based mental health services by providing an empirical road map to the processes that need to be targeted by clinical protocols (Garber, Korelitz, & Samanez-Larkin, 2012; Ialongo et al., 2006). Yet, despite this translational promise, most screening protocols do not assess vulnerabilities for psychological distress and predominantly focus on mental health symptoms (Lavigne, Meyers, & Feldman, 2016; Wissow et al., 2013). Broadening our protocols to include risk factors for psychopathology could improve prognostic models for pediatric mental health issues.

The goal of the present study was to help develop a feasible, multi-indicator approach to adolescent depression screening. Because of the prevalence, chronicity, and consequences of pediatric depression, routine depression screening is recommended by age 12 (Forman-Hoffman et al., 2016). These screening programs are tasked with not only identifying concurrent distress and functioning but also forecasting future depression risk. However, a paucity of applied studies assess prospective outcomes (Wissow et al., 2013), and basic research studies tend to use methods and analytical plans that make their findings challenging to translate into clinical settings (Youngstrom, 2014; Youngstrom et al., 2017). We sought to address this gap in the literature by explicitly examining the clinical utility and incremental validity of cognitive vulnerability measures, beyond traditional depressive symptom inventories, when forecasting depression outcomes. Specifically we examined how well our screening solutions performed when estimating risk for concurrent and prospective episode onset, as well as first lifetime episodes of depression and recurrent major depressive episodes, two clinically relevant depression outcomes (Monroe & Harkness, 2011; Petit, Hartley, Lewinsohn, Seeley, & Klein, 2013). To our knowledge, this represents the first study to model an approach for how cognitive vulnerabilities can be used in an applied context to simultaneously target multiple concurrent and prospective depression outcomes.

A Risk Factor Approach to Depression Screening

Multi-indicator screening approaches, comprising presenting symptoms, psychosocial correlates, and underlying vulnerabilities, exist for a myriad of chronic, pediatric conditions. For forecasting pediatric obesity, screening protocols often depend on multiple sources of data including body mass index, blood pressure, lipids, and self-reported physical activity (Smith, Skow, Bodurtha, & Kinra, 2013). Evidence-based screening protocols for dental cavities involve a physical exam, along with familial and environmental predictors of oral health care (Divaris, 2016). More proximal to mental health, large screening initiatives (i.e., the PROMIS initiative; Irwin et al., 2010) assess social health in youth. These measures were originally used to identify youth with social impairment secondary to a chronic disease (Varni et al., 2014); however, the measure can also help identify youth at risk for psychological distress (DeWalt et al., 2015). We therefore situate our study’s aims within the context of a larger health movement that simultaneously examines manifest symptoms and underlying risk at the screening stage.

Multiple informants or inventories (De Los Reyes et al., 2015; Lavigne et al., 2016; Mash & Hunsley, 2005) have typically been used to improve pediatric mental health screening and assessment initiatives. Although these approaches better discriminate between positive and negative diagnoses by providing multiple perspectives on depression presentations, they are still largely dependent on the manifestation of concurrent symptoms and may be limited in predicting prospective depression episodes. To date, few studies have explicitly examined incremental validity of multi-informant depression assessment procedures (Johnston & Murray, 2003), let alone investigated vulnerabilities within a translational science context. Seely, Stice and Rohde (2009) found that poor academic functioning improved our ability to predict prospective depressive episodes in female adolescents beyond self-reported depressive symptoms. Meanwhile, Cohen and colleagues (Cohen et al., 2016) found that predisaster mental health symptoms and trauma exposure best predicted postdisaster adolescent depression. We build on these studies by focusing on cognitive vulnerabilities, as they (a) are valid risk factors for adolescent depression (Cohen et al., 2017; Hankin, Snyder, & Gulley, 2016; Ingram, Miranda, & Segal, 2006), (b) can easily be screened for in applied settings, and (c) are targeted by cognitive-behavioral preventative interventions (Garber et al., 2012).

Cognitive Vulnerability for Adolescent Depression

Depressogenic cognitive vulnerabilities are defined as a stylistic, enduring way of thinking that precede the emergence of depression (Ingram et al., 2006). Early adolescence represents a critical period for these vulnerabilities to develop (Jacobs, Reinecke, Gollan, & Kane, 2008), which helps explain the heightened sensitivity to depression during this developmental epoch, especially for girls (Hankin & Abramson, 2001). Over the years, several theoretical models have articulated specific examples of these cognitive styles. The hopelessness theory of depression posits that stable and global depressogenic inferences about the cause of a negative event (i.e., attributional style), consequences of a negative event, and the implications of the event for oneself lead to the development of hopelessness and subsequent depression (Abramson, Metalsky, & Alloy, 1989). Although certain depressogenic inferential styles begin forming in childhood (e.g., depressogenic inferential styles about consequences; Cohen, Young, & Abela, 2012), evidence shows that these cognitive risks stabilize into relatively traitlike vulnerabilities and reliably predict later depression by early adolescence (Carter-Smith & Garber, 2011; Cole et al., 2008). Another cognitive vulnerability model is Beck’s (1983) theory of depression, which posits the role of a negative cognitive triad consisting of dysfunctional attitudes concerning the self, the world, and the future. Elevated levels of dysfunctional attitudes predict prospective depressive symptoms in adolescent samples (Abela & Sullivan, 2003). Finally, the response styles theory (Nolen-Hoeksema, 1991) highlights the influence of rumination in contributing to depression. Rumination involves dwelling on the potential meaning, causes, and consequences of one’s problems, concerns, or symptoms of distress. In adolescence, rumination is associated with depression episode onset and longer depressive episodes (Abela & Hankin, 2011).

Applied Developmental Psychopathology

The aforementioned studies were based on multiwave, prospective studies, the ideal approach for demonstrating the longitudinal associations between vulnerabilities and prospective outcomes. Accompanying these study designs were complex analytic plans, which utilized hierarchical linear (Singer & Willett, 2003) or structural equation modeling (Cheong, MacKinnon, & Khoo, 2003) to test the depressogenic influence of these vulnerabilities over time. Although these statistical methods are important to adequately test developmental theories of psychopathology, the intention of these analyses is not to aid clinical decision making (Hunsley & Meyer, 2003; Youngstrom, 2014). For these findings to reach their translational promise, a receiver operating characteristic (ROC) approach may be necessary.

The ROC curve is a representative plot of the true positive rate against the false positive rate across a continuum of scores, allowing one to calculate the sensitivity (i.e., the ability to correctly identify a true positive) and specificity (i.e., the ability to correctly identify a true negative) for specific cutoff points on an index test. In addition to the ROC curve, diagnostic likelihood ratios (Straus, Richardson, Glasziou, & Haynes, 2011) can help estimate the likelihood of whether an individual is presenting with, or will develop, the target disorder based on his or her scoring profile. Knowing the posterior probability of developing a target disorder allows providers and institutions to conduct their own cost–benefit analysis when making referral decisions based on screening profiles. Utilizing the ROC curve together with diagnostic likelihood ratios is viewed as a “best practice” for determining the clinical utility of a potential index test for a pediatric mental health disorder (Youngstrom, 2014; Youngstrom et al., 2017). To date, ROC approaches have been used to evaluate assessments for pediatric anxiety disorders (Van Meter et al., 2014), bipolar disorder (Youngstrom, Genzlinger et al., 2015), attention deficit hyperactivity disorder (Jarrett, Van Meter, Youngstrom, Hilton, & Ollendick, 2016) and posttraumatic stress disorder (You, Youngstrom, Feeny, Youngstrom, & Findling, 2017). However, few studies have explicitly used this approach to assess the incremental validity of including risk factors, as well as symptom measures, for predicting pediatric mental health disorders (see Cohen et al., 2016; Danielson et al., 2017, for exceptions).

In addition to the analytic plans used in developmental psychopathology, the methods typically used in this research may also inhibit the translation of these findings into applied settings. First, the majority of studies examine cognitive vulnerability in isolation (Hankin et al., 2016), preventing inferences to be made about which cognitive style may offer the strongest signal for depression risk. Examining multiple cognitive vulnerabilities simultaneously can help determine which indicators should be prioritized at the screening stage. Second, most studies concerning adolescent cognitive vulnerability have used depression symptom measures as their main outcome (Cohen et al., 2017; Hankin et al., 2016). Although youth depression is dimensionally structured at the latent level (Hankin et al., 2017), depression diagnoses still serve a critical function within the realm of clinical decision making and are recommended to use in tests of incremental validity for clinical screening and assessment protocols (Garb, 2003; Youngstrom, 2014). Important to note, the analytic approach just described can still capture dimensional nuances of depression by estimating both subthreshold and threshold risk outcomes.

Concurrent Depression, Prospective Depression, First Episodes, and Recurrent Depression

All depression outcomes are not created equal. The link between cognitive vulnerability and depression risk can vary based on past and current experiences with depression (Alloy et al., 2006). Relatedly, other findings suggest that risk profiles may differ for specific depression outcomes. For instance, first lifetime episodes of depression (FLED) may be more sensitive to environmental stressors, with subsequent episodes being triggered more by intrapersonal processes (Monroe & Harkness, 2011). To attenuate the elevated prevalence rates in adolescence, greater attention should be paid to FLED in prevention research (Allen, Hetrick, Simmons, & Hickie, 2007). Similarly, unique risk factors for a chronic pattern of depression (i.e., recurrent major depressive disorder [rMDD]) may exist (Monroe & Harkness, 2011; Petit et al., 2013). These findings have led some to suggest that prevention protocols need to develop separate screening solutions to target individuals prone to a persistent depression course (Hill, Yaroslavsky, & Pettit, 2015). Given that the risk profile for depression may differ for these specific depression outcomes, our algorithms for interpreting cognitive vulnerability measures within a screening protocol may vary when attempting to predict a concurrent or prospective episode, FLED, or rMDD. Distinguishing between concurrent and prospective outcomes, as well as more nuanced FLED and rMDD outcomes, can lead to a clearer understanding of the incremental validity of cognitive vulnerability measures by using a more homogeneous depression criterion.

The present study aimed to identify (a) the incremental validity of cognitive vulnerabilities in forecasting depression; (b) delineating which styles confer the greatest risk; and (c) optimizing screening solutions for concurrent episodes, prospective episodes, FLED, and rMDD. Given well-documented sex and age differences for depression onset and risk (Hankin & Abramson, 2001), we examined how these solutions may vary for girls versus boys and for early adolescents versus middle adolescents. By simultaneously comparing different cognitive vulnerability measures, assessing across four clinically significant depression outcomes, and applying a state-of-the-science translational analytical approach, we sought to create cognitive vulnerability-based algorithms that could improve empirically based clinical decision making for depression prevention at the screening stage.

METHOD

Participants and Procedures

A multisite sample drawn from urban and suburban communities was recruited for the present study. Participating families responded to letters sent home by local schools describing a longitudinal study focused on different predictors of emotional and behavioral well-being in youth. At baseline, the sample consisted of 473 youth who were in the sixth grade (Age_M = 11.75, Age_SD = 0.70) and ninth grade (Age_M = 14.65, Age_SD = 0.60). Adolescents were eligible for the study if parents reported during a telephone screen that their child was in the sixth or ninth grade at baseline; was fluent in English; and did not have an autism spectrum diagnosis, psychotic disorder, or intellectual disability. In our study, sixth-grade students were considered early adolescents and ninth-grade students were referred to as middle adolescents. Youth were relatively balanced with regard to sex (female = 57%) and grade (sixth = 52%). The study’s racial/ethnic composition was as follows: White = 61%, African American = 12%, Asian American = 9%, Hispanic = 7%, which is comparable to the ethnic and racial characteristics of the United States, with the exception of fewer Hispanic participants (see Hankin et al., 2015, for further details).

The caretaker and youth visited the laboratory for an in-person, in-depth assessment at baseline. Youth completed measures of cognitive vulnerability and depressive symptoms. Diagnostic interviews were conducted with both the adolescent and caregiver using a semistructured diagnostic interview for youth lifetime and current depressive episodes. Subsequently, for the next 3 years, youth and caregivers were interviewed every 6 months to ascertain whether the adolescent had a depressive episode onset in the preceding 6 months. These frequent assessments limit biases with retrospective recall (Compton & Lopez, 2014; Costello & Erkanli, 2006). In accordance with the Standards for Reporting of Diagnostic Accuracy Group statement (Bossuyt et al., 2003), diagnosticians were not privy to the scores on our indicators. Retention rate from baseline to 36-month follow-up for the study was 93%. Caretakers provided informed written consent for their own and their child’s participation; youth provided written assent. The Institutional Review Boards at both study sites approved all procedures.

Measures

Depression Diagnoses

Trained interviewers administered the Mood Disorders section of the Schedule for Affective Disorders and Schizophrenia for School-Age Children (K-SADS-PL; Kaufman et al., 1997) to youth and caretakers to assess for pediatric depression at baseline and each follow-up. At baseline, youth and caregivers reported lifetime history of depression and current diagnostic status, whereas at each follow-up, families reported on episode occurrence over the preceding 6 months. Licensed clinical psychologists trained the interviewers to conduct the diagnostic interviews. Both interviews informed youths’ diagnostic status using best estimate diagnostic procedures (Klein, Dougherty, & Olino, 2005). Discrepancies between parent and child reports were resolved during weekly supervision meetings with a licensed clinical psychologist based on the quality of the diagnostic report (e.g., behavioral specific examples were provided for a given symptom) and the empirical literature (e.g., self-report may be better for the internalizing aspects of adolescent depression; De Los Reyes et al., 2015). Diagnostic interview interrater reliability was good (K = .91) based on approximately 20% of reviewed interviews. Youth were diagnosed with a depressive episode if they met Diagnostic and Statistical Manual for Mental Disorders (4th ed.; American Psychiatric Association, 1994) criteria for major depressive disorder–definite, major depressive disorder–probable (four depressive symptoms for at least 2 weeks), or minor depressive disorder–definite (two or three threshold depressive symptoms for at least 2 weeks).

In the present study, four binary depression outcomes were calculated from the Schedule for Affective Disorders and Schizophrenia for School-Age Children: (a) concurrent episodes (i.e., presenting with a depressive episode at baseline), (b) prospective episodes (i.e., a depressive episode during the follow-up period), (c) FLED (i.e., at least one episode during the study with no prior lifetime diagnosis reported at baseline), and (d) rMDD (i.e., multiple depressive episodes during the course of the study or a lifetime history of depression reported at baseline and at least one depressive episode reported during the study).

The Children’s Depression Inventory

The Children’s Depression Inventory (CDI; Kovacs, 1992), a self-report 27-item questionnaire, assessed pediatric depressive symptoms. The CDI was chosen because it is the most commonly used measure of youth depression (Myers & Winters, 2002) and a recommended measure for assessing depression in applied settings (Klein et al., 2005). For the present study, the CDI ranged from 0 to 35 (M = 7.08, SD = 5.87 at baseline) and demonstrated adequate reliability (α = 0.84).

Children’s Response Style Questionnaire–Rumination Subscale

The Children’s Response Style Questionnaire–Rumination Subscale (CRSQ-R; Abela, Vanderbilt, & Rochon, 2004) is modeled after Nolen-Hoeksema’s Response Style Questionnaire (Nolen-Hoeksema & Morrow, 1991). The CRSQ-R is a 13-item self-report measure that assesses one’s tendency to ruminate or focus on negative aspects of oneself. For each item, youth indicate how often they respond in a ruminative way when feeling sad, with higher scores indicating a greater tendency to ruminate. The CRSQ-R is a reliable and valid measure of rumination in youth samples (Abela & Hankin, 2011). The CRSQ-R had a Cronbach’s alpha level of 0.79 in the present study.

Adolescent Cognitive Style Questionnaire

The Adolescent Cognitive Style Questionnaire (ACSQ; Hankin & Abramson, 2002) is a self-report inventory that measures inferences about cause, consequence, and oneself, as featured in hopelessness theory. The ACSQ presents the adolescent with negative hypothetical events in achievement and interpersonal domains and asks the youth to make inferences about the causes (internal–external, stable–unstable, and global–specific) and consequences of the event and the characteristics about the self based on the hypothetical event. Each item dimension is rated from 1 to 7, with higher scores indicating a more negative cognitive style. The ACSQ has demonstrated excellent internal consistency, reliability, good test–retest reliability, and a factor structure consistent with hopelessness theory (Hankin & Abramson, 2002). Internal reliability in this sample was 0.92.

Children’s Dysfunctional Attitude Scale

The Children’s Dysfunctional Attitude Scale (CDAS; Abela & Sullivan, 2003) is a questionnaire designed to assess dysfunctional attitudes in youth. For each item (e.g., “I should be good at everything I try”), participants are asked to rate how much each statement applies to them (i.e., never true, sometimes true, most of the time true, and always true). In the present study, we utilized a short form (20 items) of the CDAS (Flouri & Panourgia, 2014). Total scores on the measure range between 0 to 60, with higher scores indicating increased dysfunctional attitudes. The CDAS has adequate reliability and predictive validity of concurrent and prospective adolescent depressive symptoms (Flouri & Panourgia, 2014; McWhinnie, Abela, Knäuper, & Zhang, 2009). Internal reliability in this sample was 0.81.

Data Analytic Strategy

Baseline scores on the CDI and all cognitive vulnerabilities (CRSQ, ACSQ, and CDAS) represented our main predictors for our four binary outcomes: (a) a concurrent episode at baseline, (b) a prospective episode during the follow-up period, (c) FLED, and (d) rMDD. Initially, two-way interactions were created in hierarchical logistic regression models to examine if cognitive vulnerabilities’ association with depression varied as a function of sex and/or grade. If significant, area under the curves (AUCs) were computed for these subpopulations separately. Next, we used the “best approach” ROC steps outlined by Youngstrom (2014) to determine the validity of an index test. For ROC analyses, significance is determined if the AUC does not include 0.50 in the confidence interval; however, higher cutoffs for clinical utility have been recommended. In the present study, an AUC greater than 0.56 conferred a significant, albeit small effect (Rice & Harris, 2005), whereas an AUC of 0.70 (Swets, 1988) was prioritized. Hanley and McNeil’s (1983) method was used to examine significant differences between the AUCs. Finally, diagnostic likelihood ratios (DLRs; Straus et al., 2011) were calculated for each inventory. DLRs were based on informative tertiles with the cutoff for the subthreshold group placed at 70% sensitivity and the high-risk group being formed at 90% specificity for forecasting prospective episodes of depression. These cutoffs were based on the approximate cutoffs of current screening initiatives using symptom-based measures for pediatric mental health conditions (Lavigne et al., 2016).

We repeated variants of these steps to test for incremental validity. First, we used CDI scores to predict each cognitive vulnerability and saved the residuals. These residual scores represent the independent variance of the cognitive vulnerability beyond depressive symptoms. We then examined if the AUC for the residual score remained significant (see Edens, Skeem, & Douglas, 2006; Hastings, Krishnan, Tangney, & Stuewig, 2011, for examples of using this residual approach to calculate an adjusted ROC curve). If multiple cognitive vulnerabilities remained significant within this approach, we examined the residuals from these cognitive vulnerabilities to ensure their effect was unique (e.g., we examined the difference between the observed and predicted scores for rumination based on the attributional style score). Finally, DLRs for informative tertile scores across indicators were summed to determine which combination of risk factor, and at what level, conferred the greatest risk. This allowed us to calculate the DLR for both convergent (e.g., high depressive symptoms and high rumination) and divergent (e.g., high depressive symptoms and low rumination) profiles.

RESULTS

Preliminary Analyses

Table 1 presents the correlations and descriptive statistics for our baseline predictors. With regard to our criterion variables, all of our outcomes exceeded the minimum number of cases needed to conduct ROC (N = 20; Kraemer, 1992): (a) 4.5% (N = 21) had a concurrent depressive episode at baseline, (b) 30% (N = 137) of our sample had a prospective depressive episode during the follow-up stage, (c) 22.6% (N = 77) had a FLED, and (d) 18.9% (N = 84) had rMDD. Regression analyses suggested that the relation between attributional style and rMDD varied as a function of age (B = .22, SE = .10, Wald = 4.54, p = .03), such that attributional style forecasted rMDD in ninth grade (p < .01) but not sixth grade students (p > .05). Thus, we calculated AUCs separately for sixth- and ninth-grade students with models that included attributional style forecasting rMDD. All other associations between cognitive vulnerability and depression were invariant to sex and age (p > .05).

TABLE 1.

Descriptive Statistics and Bivariate Correlations between Baseline Predictors, Age, and Sex

	1	2	3	4	5	6
1. Rumination (M = 26.39, SD = 7.87)
2. Dysfunctional Attitudes (M = 33.25, SD = 7.43)	.44**
3. Attributional Style (M = 89.11, SD = 26.27)	.49**	.46**
4. Depressive Symptoms (M = 7.29, SD = 5.90)	.56**	.43**	.47**
5. Sex	.12**	−.07	.02	.05
6. Age	.13**	.19**	.24**	.22**	−.02

Note: Correlations with sex are point biserial correlation coefficients, while all other correlations are Pearson correlation coefficients. Rumination = Total score on the Children’s Response Style Questionnaire–Rumination subscale (Abela et al., 2004); Dysfunctional Attitudes = total score on the Children’s Dysfunctional Attitude Scale (Abela & Sullivan, 2003); Attributional Style = total score on the Adolescent Cognitive Style Questionnaire (Hankin & Abramson, 2002); Depressive Symptoms = total score on the Children’s Depression Inventory (Kovacs, 1992); Sex = parent-reported child sex (male = 0, female = 1); Age = parent-reported child’s chronological age.

^**p < .01.

Area Under the Curve

AUCs for our depression outcomes are presented in Table 2. For concurrent episodes at baseline, only rumination and depressive symptoms conferred diagnostic status. For prospective episodes, AUCs for all indicators were significant and exerted a medium effect. We found a similar effect for all predictors of rMDD, with the exception of attributional style not being significant in early adolescents. As for FLED, only rumination and dysfunctional attitudes were significant. Pairwise comparisons using Hanley and McNeil’s (1983) method suggested that across analyses, significant indicators forecasted diagnostic patterns similarly (p > .05; e.g., the AUCs for rumination and depressive symptoms predicting concurrent episodes, prospective episodes, and rMDD were not significantly different).

TABLE 2.

AUC for Individual Predictors for Depression Outcomes

	AUC	SE	Cohen’s d (Effect Size)
Concurrent Episodes
Rumination	0.69*	0.07	0.70 (Medium)
Dysfunctional Attitudes	0.56	0.07	0.21 (Small)
Attributional Style	0.56	0.07	0.21 (Small)
Depressive Symptoms	0.70*	0.06	0.74 (Medium)
Prospective Episodes
Rumination	0.67* (unique predictor)	0.03	0.62 (Medium)
Dysfunctional Attitudes	0.62*	0.03	0.43 (Medium)
Attributional Style	0.64* (unique predictor)	0.03	0.51 (Medium)
Depressive Symptoms	0.63* (unique predictor)	0.03	0.47 (Medium)
FLED
Rumination	0.63* (unique predictor)	0.03	0.47 (Medium)
Dysfunctional Attitudes	0.58*	0.04	0.29 (Small)
Attributional Style	0.56	0.04	0.21 (Small)
Depressive Symptoms	0.56	0.06	0.21 (Small)
rMDD
Rumination	0.69* (unique for 6th grade only)	0.03	0.70 (Medium)
Dysfunctional Attitudes	0.63*	0.04	0.47 (Medium)
Attributional Style
(6th grade/9th grade)	0.54/0.67* (unique predictor)	0.06/0.05	0.14/0.62 (Medium)
Depressive Symptoms	0.66* (unique predictor)	0.04	0.58 (Medium)

Note: For rumination forecasting rMDD, the AUC reflects the curve for the total sample but is only a unique predictor in the sixth grade. AUC = area under the curve; SE = standard error for AUC; Cohen’s d = effect size; Small, Medium, and Large = effect size for predictor (Rice & Harris, 2005); Concurrent episodes = meeting criteria for an episode of depression at baseline; Prospective episodes = meeting criteria for an episode of depression during the follow-up period; FLED = first lifetime episode of depression (i.e., episode of depression during 36 months with no lifetime history); rMDD = recurrent major depression disorder (i.e., more than one episode of depression over the lifetime); Rumination = total score on the Children’s Response Style Questionnaire–Rumination subscale (Abela et al., 2004); Dysfunctional attitudes = total score on the Children’s Dysfunctional Attitude Scale (Abela & Sullivan, 2003); Attributional style = total score on the Adolescent Cognitive Style Questionnaire (Hankin & Abramson, 2002); Depressive symptoms = total score on the Children’s Depression Inventory (Kovacs, 1992); Unique predictor = whether the residuals were significant once covarying out other significant predictors for that criterion (first depressive symptoms, and then other significant predictors).

^*p < .05.

We next examined the unique variance of each indicator. For concurrent episodes, neither rumination (AUC = .59, p > .10) nor depressive symptoms (AUC = .60, p > .10) were significant. This suggests that the unique variance associated with either predictor is not enough to indicate a concurrent episode. For predicting prospective episodes, rumination (AUC = 0.62, p < .001), dysfunctional attitudes (AUC = .59, p < .01), and attributional style (AUC = 0.58, p <.01), each had a unique effect beyond depressive symptoms. Dysfunctional attitudes, however, were not unique after accounting for rumination (AUC = .54, p > .10), but the residuals associated with attributional style secondary to dysfunctional attitudes and rumination were still significant (AUC = .62, p = .03). As for FLED, residuals associated with dysfunctional attitudes were not significant once accounting for rumination (AUC = .50, p > .50). For rMDD in early adolescents, only residuals associated with rumination were significant (AUC = .69, p = .001). In middle adolescents, only the effect of attributional style was independent of depressive symptoms (AUC = .60, p = .03).¹

Diagnostic Likelihood Ratios

DLRs for each unique, significant predictor can be found in the top panel of Table 3. For rumination, low scores ranged from 0 to 19, medium scores ranged from 20 to 35, and high scores were 36 and above. For attributional scores, low scores were 61 or lower, medium scores were 62 to 114, and high scores were 115 and above. Finally, for depressive symptoms, 0 to 3 corresponded to low scores, 4 to 14 were medium scores, and high scores were 15 and above. Of note, a score of 15 falls within the published recommendations range for clinical cutoffs (13–20) published by Kovacs (1992). Despite neither depressive symptoms nor rumination conferring unique variance for concurrent episodes, we still calculated DLRs for both indicators to examine the incremental contribution of rumination symptoms above and beyond depressive symptoms alone. With regard to individual measures, each significant cognitive vulnerability conferred an approximate twofold to threefold increase in risk, and depressive symptoms corresponded to a twofold to fourfold increase in risk across our depression outcomes.²

TABLE 3.

Diagnostic Likelihood Ratios for Significant and Unique Individual Predictors and Combined Models

	Low	Medium	High
Independent Models
Concurrent Episodes
Rumination	0.44	0.57	3.97
Depressive Symptoms	0.29	1.09	2.60
Prospective Episodes
Rumination	0.58	0.91	1.94
Attributional Style	0.43	0.83	2.79
Depressive Symptoms	0.73	0.87	4.30
FLED
Rumination	0.40	1.26	0.94
rMDD
Rumination (6th Grade)	0.29	1.09	2.12
Attributional Style 9th Grade)	0.22	0.76	2.89
Depressive Symptoms (6th/9th)	0.81/0.50	0.95/1.01	2.17/3.48
Combined Models
Concurrent Episodes (Rumination and Depressive Symptoms)
	0.47	0.66	3.34
Prospective Episodes (Depressive Symptoms, Rumination, and Attributional Style)
	0.59	0.88	6.67
FLED (Rumination and Depressive Symptoms)
	0.49	1.00	1.85
rMDD (Rumination/Depressive Symptoms–6th Grade/Attributional Style/Depressive Symptoms–9th Grade).
	0.40/0.00	0.95/0.64	2.36/2.98

Note: The following are cutoff scores on each individual measure as well as for the combined models. Low: Rumination (0–19), Attributional Style (29–61), Depressive Symptoms (0–3), Combined Model = two low scores on individual inventories. Medium: Rumination (20–35), Attributional Style (62–114), Depressive Symptoms (4–14). Combined Model = two medium scores or one low, one medium, and one high score or two high scores and one low score. High: Rumination (36 and up), Attributional Style (115 and up), Depressive Symptoms (15 and up), Combined Model = all high scores or two high scores and one moderate score or one moderate score and one high score. For individuals strictly interested in FLED, only the rumination measure should be used. Table 3 presents the diagnostic likelihood ratios (DLRs) for different tertiles of each screening inventory, as well as the DLRs for different combinations of these tertiles. DLRs represent the ratio of target disorders present within a specific scoring range out of the total number of target disorders divided by the number of target disorders absent within that scoring range divided by the total number of target disorders absent (see Straus et al., 2011). Concurrent episodes = depression episode at baseline; Prospective episodes = a depression episode during the follow-up assessment; FLED = first lifetime episode of depression (i.e., episode of depression during 36 months with no lifetime history); rMDD = recurrent major depression disorder (i.e., more than one episode of depression over the lifetime); Rumination = total score on the Children’s Response Style Questionnaire–Rumination subscale (Abela et al., 2004); Attributional style = total score on the Adolescent Cognitive Style Questionnaire (Hankin & Abramson, 2002); Depressive symptoms = total score on the Children’s Depression Inventory (Kovacs, 1992).

DLRs for all combined models are in the bottom of Table 3. As we did not find any significant differences between predictors, we did not distinguish profiles based on whether a particular indicator was high, medium, or low (i.e., high depressive symptoms and low rumination levels would be represented the same as someone with high rumination and low depressive symptoms). For concurrent episodes, being above threshold in the combined model corresponded to a 28% higher DLR than the DLR for high depressive symptoms. In other words, having moderate or high rumination together with high depressive symptoms led to a 28% increase in likelihood of being currently depressed. For prospective episodes, adolescents with high attributional style, rumination, and depressive symptoms (or even just two out of the three and a moderate score for the third indicator) were 6 times as likely to develop a depressive episode during the follow-up period than not. Presenting above threshold in the combined model represents an approximate 50% increase for depression risk compared to assessing depressive symptoms alone. For FLED, presenting above threshold in the combined model led to a 46% to 96% increase in DLRs above having moderate-elevated rumination scores alone.³ Finally, DLRs for rMDD models are presented separately for early and middle adolescents. DLRs associated with threshold scores on the combined model were approximately the same as those found for depressive symptoms alone. However, combined model DLRs seem to be more specific (i.e., correctly identify a nonpositive case), as low scores in the combined models led to dramatically lower DLRs compared to assessing depressive symptoms alone. In other words, the combined models for forecasting rMDD specifically improved our ability to forecast those not at risk for developing a recurrent depressive course.

DISCUSSION

The goal of the present study was to leverage the considerable amount of basic research on cognitive vulnerabilities for depression (Cohen et al., 2017; Hankin et al., 2016; Jacobs et al., 2008) into improved methods for predicting adolescent depression outcomes. Depression screening protocols have two main objectives: to identify concurrent functional impairment/distress and to forecast prospective depression risk (Forman-Hoffman et al., 2016). To date, the majority of applied research focuses on improving strategies for identifying concurrent depressive episodes with a paucity of studies investigating the prediction of prospective outcomes. We sought to address this gap in the literature by examining the incremental validity of including cognitive vulnerabilities when screening for both concurrent and prospective adolescent depression. Furthermore, we tested prediction algorithms for FLED and recurrent episodes (rMDD) separately as these convey important, significant information that has not previously been examined. Using a translational analytic approach, we found that assessing rumination and attributional style at the screening stage incrementally improves identification of depression during adolescence, especially prospective and persistent patterns of depression. Next we discuss the implications of our findings and how to integrate our results into practice.

To date, the few studies that have examined cognitive styles using an ROC approach have shown inconsistent support for their ability to identify a concurrent depression diagnosis (e.g., Shapero et al., 2015; Young & Dietrich, 2014). This pattern of findings mirrors the current study, as we received mixed support for cognitive vulnerability measures’ ability to improve identification of concurrent episodes. To add a novel indicator to a screening battery, the inventory must demonstrate added, unique variance in predicting the target disorder beyond the existing protocol (Garb, 2003; Hunsley & Meyer, 2003). In our study, the AUCs for attributional style and dysfunctional attitudes were not significant for current depression identification. The AUC for rumination, although statistically significant, did not differ from depressive symptoms and did not demonstrate unique variance when predicting concurrent depressive episodes. At the same time, a 28% increase in DLRs for those with elevated rumination and depressive symptoms, compared to the DLR for only depressive symptoms, suggests a potential meaningful difference in one’s likelihood for presenting with a current depression diagnosis. Some argue that standard AUC benchmarks for incremental validity may underestimate the impact of a novel indicator (Pencina, Steyerberg, & D’Agostino, 2011), so alternative approaches may be necessary to determine whether rumination should ultimately be used in screening and assessment models for identifying concurrent depression.

The value of using a multi-indicator approach was best exemplified when predicting prospective depressive episodes. Rumination and attributional style both improved our ability to predict prospective depressive episodes in early and middle adolescents. DLRs for the at-risk category in the combined model were more than 50% greater than the DLRs associated with using depressive symptoms alone. The challenges of using single inventories to determine concurrent depression diagnostic status (Fristad, Emery, & Beck, 1997; Klein et al., 2005; Matthey & Petrovski, 2002) seem to be exacerbated when examining future distress. In our study no single indicator, including depressive symptoms, for prospective episodes reached the AUC benchmark of 0.70 (Swets, 1988). Others have addressed this issue for concurrent depression by querying multiple informants (De Los Reyes et al., 2015) or utilizing multiple mental health screening inventories (Lavigne et al., 2016). For prospective episodes, using a multi-informant or multimental health screening approach may be limited, as the algorithms are largely dependent on current or recent symptoms. Instead, others have recommended incorporating risk factors into our evidence based assessment approaches (Youngstrom et al., 2017). Although these risk factors tend to focus on immutable demographic factors (e.g., sex), we demonstrated that the assessment of cognitive vulnerabilities led to a significant improvement in our ability to forecast prospective depression episodes when compared to using depression symptoms as the sole indicator.

Our study represented one of the first applications of a translational analytic approach to develop screening solutions for FLED and rMDD in adolescence. Prevention research has recommended an increased focus on both of these specific depression outcomes (Allen et al., 2007; Petit et al., 2013). For FLED, rumination was the only significant indicator; however, its translation into a screening framework is not straightforward. Deflated DLRs, compared to our other outcomes, suggests that our algorithm did not calibrate as well for FLED (Straus et al., 2011). Therefore, it may be advantageous to consider other risk factors beyond depressive symptoms and cognitive vulnerability when creating protocols for this depression outcome. For instance, major stressful life events may play a larger role in FLED compared to recurrent episodes (Monroe & Harkness, 2011), and more proximal indicators of our stress response symptom (e.g., HPA-Axis) may be instrumental in forecasting an initial depressive episode (Mazurka, Wynne-Edwards, & Harkness, 2016). Thus, future research may need a multi-indicator protocol that combines rumination, stressors, and biological indicators to adequately predict FLED.

Finally, for rMDD, we found that pairing rumination with depressive symptoms best forecasted this persistent course in early adolescence, and attributional style and depressive symptoms best identified rMDD in middle adolescence. Interestingly, the value of a multi-indicator approach within the context of rMDD seems to be that it is more specific, as opposed to more sensitive, for predicting positive cases. DLRs in the lowest risk group either exceeded or came close to the 0.25 threshold posited by Straus and colleagues (2011) in which one could be “moderately certain” of a negative outcome (i.e., no diagnosis). These estimates were dramatically lower than the DLRs reported for the lowest tertile of depressive symptoms alone. Cognitive vulnerability measures therefore may best be used in models for persistent adolescent depression (Hill et al., 2015) to help rule out individuals not at risk for rMDD so that resources can be better prioritized for those who are most vulnerable to this chronic depression course.

The present study should be viewed in light of several limitations. First, it is important to acknowledge that although cognitive vulnerabilities and depressive symptoms are conceptually distinct at the latent level (Hankin et al., 2016; Jacobs et al., 2008), in the present study they were both assessed via self-report. It is critical for future research to use a multimethod approach when attempting to identify novel predictors to reduce the shared method variance between screening indicators (see De Los Reyes & Aldao, 2015, for a discussion). Second, diagnostic outcomes were based on clinician “best estimates” from multi-informant diagnostic interviews. Although this approach reflects best practice (Klein et al., 2005), clinicians may be more swayed by parental compared to youth report (Youngstrom et al., 2011) leading some to suggest that diagnostic interviews should be viewed as a “fuzzy” gold standard (Zhou, Obuchowski, & McClish, 2002). Third, cutoff scores on dimensional measures have inherent limitations, especially for those at or near the threshold who may be improperly classified (Sheldrick et al., 2015). Although we did our best to mitigate this limitation by using multilevel diagnostic likelihood ratios (Straus et al., 2011), we acknowledge that our multiple cutoff scores are vulnerable to misclassification. Fourth, we only assessed for mood disorders and did not include other pediatric diagnoses. Given high rates of comorbidity between depression and other pediatric disorders (e.g., anxiety; Avenevoli, Swendsen, He, Burstein, & Merikangas, 2015; Cohen, Young, Gibb, Hankin, & Abela, 2014), as well as the association between cognitive vulnerabilities and discrete forms of psychological distress (Hankin et al., 2016), future research should examine the sensitivity of our findings with regard to depression. Finally, our study was conducted within the context of a research study. It is important that future studies replicate these findings in applied settings, as demand characteristics may influence how adolescents respond within a research study compared to a clinical context (Krosnick, 1999). Under a recent rubric put forth to evaluate empirically based assessment protocols the current study qualifies as “adequate” for demonstrating clinical utility of an index test for pediatric mental health but would need to be replicated in an applied setting to merit a higher rating (Youngstrom et al., 2017).

Clinical Implications

Across medicine (Bossuyt et al., 2003), and within pediatric mental health (Youngstrom et al., 2017), there is a movement to make basic research more accessible to clinical decision making. Table 4 provides four example screening profiles and how the approach modeled in the present study can be used to improve empirically based referral decisions. It is our hope that modeling these examples will not only facilitate the use of cognitive vulnerability measures into practice but also show the potential for applying this translational analytic plan to other risk factors for adolescent mental health.

TABLE 4.

Stoplight Model for Examples of Screening Cases

Exemplars	Pretest Prob	Score Profile	DLR	Posttest Prob
6th-Grade Boy	Concurrent: 2.0%	CDI: 15 (Hi)	Concurrent: 3.34	Concurrent: 6.4%
	Prospective: 16.0%	CRSQ: 29 (Med)	Prospective: 0.88	Prospective: 14.4%
	FLED: 10.3%	ACSQ: 98 (Med)	FLED: 1.85	FLED: 17.1%
	rMDD: 9.4%		rMDD: 2.36	rMDD: 18.9%
6th-Grade Girl	Concurrent: 3.4%	CDI: 15 (Hi)	Concurrent: 3.34	Concurrent: 9.4%
	Prospective: 26.6%	CRSQ: 26 (Med)	Prospective: 6.67	Prospective: 70.1%
	FLED: 26.2%	ACSQ: 124 (Hi)	FLED: 1.85	FLED: 40.0%
	rMDD: 15.0%		rMDD: 2.36	rMDD: 29.4%
9th-Grade Boy	Concurrent: 6.3%	CDI: 15 (Hi)	Concurrent: 3.34	Concurrent: 17.6%
	Prospective: 30.1%	CRSQ: 29 (Hi)	Prospective: 6.67	Prospective: 74.1%
	FLED: 16.9%	ACSQ: 92 (Med)	FLED: 1.85	FLED: 27.5%
	rMDD: 16.1%		rMDD: 2.98	rMDD: 56.8%
9th-Grade Girl	Concurrent: 6.6%	CDI: 14 (Med)	Concurrent: 0.66	Concurrent: 4.7%
	Prospective: 45.5%	CRSQ: 23 (Med)	Prospective: 0.88	Prospective: 42.8%
	FLED: 33.8%	ACSQ: 131 (Hi)	FLED: 1.00	FLED: 33.8%
	rMDD: 33.3%		rMDD: 2.98	rMDD: 59.8%
Interpretation
6th-Grade Boy	Yellow Zone: Despite an elevated DLR for a concurrent episode, the likelihood of presenting with a current episode are still quite low based on the post-test probability. Absent any critical symptoms, an immediate referral for a full assessment is not necessary due to the low risk of a current or future episode. At the same time, the elevated risk for rMDD and FLED warrants routine monitoring, and potentially even increased monitoring in the coming years.
6th-grade Girl	Red Zone: An assessment for a full mental health assessment is warranted and any available preventative services should be initiated immediately based on these screening scores. It is particularly alarming that early adolescent girls with this scoring profile have an over 70% likelihood of developing a depressive episode in the upcoming 3 years despite the fact that less than 10% with this profile will be currently depressed.
9th-Grade Boy	Red Zone: Middle adolescent boys with this scoring profile should immediately be referred for a mental health assessment and preventative services should be delivered. These youth are nearly three times more likely to be experiencing a concurrent depressive episode than the base rate, and are at significant risk for a prospective episode. The heightened risk for rMDD is another reason that an immediate referral is warranted.
9th-Grade Girl	Yellow Zone: Middle adolescent girls with this profile should be closely monitored, but a referral for a full assessment may not be imminent. Compared to their peers, youth with this scoring profile are at decreased or equivalent risk for current, future, and FLED depression outcomes. Yet, increased monitoring is warranted despite a subthreshold CDI score (14) due to these youth’s heightened risk for rMDD. Increased monitoring may help identify risk for this pattern earlier allowing for an opportunity to prevent a chronic depression course.

Note: Exemplars = examples of scoring profiles based on demographic data and the scoring profile; Pretest Prob = percentage chance of each depression outcome based on sex and age; Score Profile = sample scores on significant, unique indicators; DLR = diagnostic likelihood ratio; Posttest Prob = (prevalence/ (1-prevelance) × DLR) / ((prevalence/(1 – prevelance)) + 1) (Straus et al., 2011); Concurrent = current depression diagnosis at baseline; Prospective = experiencing a depressive episode during the follow-up period; FLED = first lifetime episode of depression (i.e., episode of depression during 36 months with no lifetime history); rMDD = recurrent major depression disorder (i.e., more than one episode of depression over the lifetime); Rumination = total score on the Children’s Response Style Questionnaire–Rumination subscale (Abela et al., 2004); Attributional style = total score on the Adolescent Cognitive Style Questionnaire (Hankin & Abramson, 2002); Depressive symptoms = total score on the Children’s Depression Inventory (Kovacs, 1992); Hi = high tertile; Med = medium tertile; Yellow Zone = increased monitoring; Red Zone = refer for mental health services.

Understanding the base rate for the target disorder is important for protecting against the base rate fallacy (Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz, & Woloshin, 2007) and allowing protocols to develop their own decision rules based on objectives and resources (Sheldrick et al., 2015; Youngstrom, 2014). Due to the increased risk for depression in girls compared to boys and middle adolescents compared to early adolescents (Avenevoli et al., 2015; Hankin & Abramson, 2001), we calculated our sample’s base rate for our four main depression outcomes based on these demographic characteristics (Columns 1 and 2). Although we decided to use base rates from our own study for these examples, epidemiological studies (see Schaefer et al., 2017) can also be used to determine pretest probabilities. In the third column in Table 4, we selected scores at or approaching the cutoff (15) for the CDI identified in the current study. These CDI scores were selected due to scores at the threshold being especially challenging from a referral perspective (Sheldrick et al., 2015). Use of additional indicators of depression may be particularly beneficial in these situations.

The fourth column represents the tailored DLRs for each depression outcome derived from Table 3, and the posttest prevalence is presented in the fifth column (see Straus et al., 2011, for calculation details). Within an empirically based assessment approach, referral decisions may ultimately be based on the posttest odds/prevalence for developing the target disorder (Youngstrom, 2014). For the present study, we used an adapted version of Youngstrom, Choukas-Bradley, Calhoun, and Jensen-Doss’s (2015) stoplight model for the assessment context. For our study, the “yellow zone” represents increased monitoring and the “red zone” reflects the need to refer for a full assessment and initiation of available preventative services. We note the differences between the yellow zone and red zone are highly context dependent and ultimately depend on the objectives of the screening program and the resources available for follow-up services (Youngstrom et al., 2017).

Interpretations for these four example cases are detailed in the bottom half of Table 4. These examples collectively illustrate the potential importance of accounting for cognitive vulnerabilities when making referral decisions and assessing multiple depression outcomes. For instance, in the first two examples, an early adolescent boy and early adolescent girl have identical depression scores. Even after accounting for differences in pretest probability, and using the higher DLR derived from the combined model (3.34) compared to using depressive symptoms alone (2.60), one would still probably decide to handle referral decisions the same when only using concurrent episodes as the criterion. However, examining the contents of Table 4 it becomes clear that concern should be heightened in the sixth-grade girl, particularly in relation to prospective diagnoses. Specifically, her probability of having a depressive episode over the next 3 years is over 70%, approximately 5 times greater than her male counterpart due in large part to her elevated attributional style score. Recognition of these differing recommendations for these two screening exemplars is possible only by distinguishing between depression outcomes and including cognitive vulnerability measures at the screening stage.

Finally, in addition to making individual patient decisions, our findings can have practical consequences in the designs of emerging preventative models for depression. Using an evidence-based medicine approach to depression screening can advance stepped-care depression prevention approaches in unselected adolescent samples (Van Straten, Hill, Richards, & Cuijpers, 2015). Stepped-care models are guided by the principles that patients may respond to lower levels of intervention prior to being sent for therapy and that attempting these low-cost interventions (e.g., online resources; Cuijpers, Van Straten, & Andersson, 2008) may be sufficient. Stepped-care and associated multistage screening protocols (Lavigne et al., 2016; Morey, Arora, & Stark, 2015) are also advantageous due to their inherent, dimensional approach to operationalizing psychopathology. To date, most stepped-care models have focused on adults, have used depression symptoms as the screen at each stage, and have not focused on prospective risk (Rohde, 2015; Van Straten et al., 2015). Implementing our findings into screening protocols for stepped-care models for adolescent depression can help allocate resources based on prospective risk and specific depression outcomes (e.g., rMDD) as one enters the vulnerable adolescent years.