Skip to main content
PMC home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2011 Nov 15;21(1):41–51. doi: 10.1002/mpr.357

Validation of diagnoses of distress disorders in the US National Comorbidity Survey Replication Adolescent Supplement (NCS‐A)

Jennifer Greif Green 1, Shelli Avenevoli 2, Michael J Gruber 3, Ronald C Kessler 3,✉, Matthew D Lakoma 3, Kathleen Ries Merikangas 4, Nancy A Sampson 3, Alan M Zaslavsky 3
PMCID: PMC3402028  NIHMSID: NIHMS358845  PMID: 22086845

Abstract

Research diagnostic interviews need to discriminate between closely related disorders in order to allow comorbidity among mental disorders to be studied reliably. Yet conventional studies of diagnostic validity generally focus on single disorders and do not examine discriminant validity. The current study examines the validity of fully‐structured diagnoses of closely‐related distress disorders (generalized anxiety disorder, post‐traumatic stress disorder, major depressive episode, and dysthymic disorder) in the lay‐administered Composite International Diagnostic Interview Version 3.0 (CIDI) with independent clinical diagnoses based on the Schedule for Affective Disorders and Schizophrenia for School‐Age Children (K‐SADS) in the US National Comorbidity Survey Replication Adolescent Supplement (NCS‐A). The NCS‐A is a national survey of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV) among 10,148 adolescents. A probability sub‐sample of 347 of these adolescents and their parents were administered blinded follow‐up K‐SADS interviews. Good concordance [area under the receiver operating characteristic curve (AUC)] was found between diagnoses based on the CIDI and the K‐SADS for generalized anxiety disorder (AUC = 0.78), post‐traumatic stress disorder (AUC = 0.79), and major depressive episode/dysthymic disorder (AUC = 0.86). Further, the CIDI was able to effectively discriminate among different types of distress disorders in the sub‐sample of respondents with any distress disorder. Copyright © 2011 John Wiley & Sons, Ltd.

Keywords: major depressive episode, generalized anxiety disorder, post‐traumatic stress disorder, WHO Composite International Diagnostic Interview (CIDI), US National Comorbidity Survey Replication Adolescent Supplement (NCS‐A)

Introduction

In epidemiologic studies, diagnostic interviews need not only to accurately identify individual diagnoses, but also to discriminate among related disorders to provide reliable estimates of comorbidity. Typically, studies of diagnostic validity focus on a single disorder and do not examine discriminant validity among comorbid disorders. The current study takes a step in the direction of addressing this problem by examining the individual and discriminant validity of fully‐structured diagnoses of a related set of disorders in the lay‐administered Composite International Diagnostic Interview Version 3.0 (CIDI) administered in the US National Comorbidity Survey Replication Adolescent Supplement (NCS‐A). This is one in a series of papers examining CIDI disorder‐specific diagnostic validity (Green et al., 2010; Green et al., 2011) in the NCS‐A sample.

The disorders that are the focus of this paper are classified as distress disorders and include generalized anxiety disorder (GAD), post‐traumatic stress disorder (PTSD), major depressive episode (MDE), and dysthymic disorder (DYS). Several factor analytic studies have found distress disorders to be a distinct set of internalizing disorders, which differentiate from fear disorders such as panic and phobia (Clark and Watson, 2006; Krueger, 1999; Watson, 2005). Analysis of NCS‐A data supports this structure, showing that these distress disorders are commonly comorbid (Kessler et al., 2011). Here, we evaluate CIDI diagnostic validity for several distress disorders individually and also whether the CIDI is able to effectively distinguish among these distress disorders. The fully‐structured CIDI is designed to be administered by trained lay interviewers to generate diagnoses by the criteria of both the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV) and the International Classification of Diseases, 10th Version (ICD‐10) systems (Kessler and Üstün, 2004). We compare CIDI diagnoses to independent clinical reappraisal interviews with the Schedule for Affective Disorders and Schizophrenia for School‐Age Children, Present and Lifetime Version (K‐SADS‐PL; Kaufman et al., 1997), an instrument commonly used to assess DSM disorders in research studies (Brooks and Kutcher, 2001). Other reappraisal studies of lay‐administered interviews for distress disorders have documented weak to moderate agreement with clinician interviews and we are interested in whether the same modest validity appears in the NCS‐A (Ezpeleta et al., 1997; Schwab‐Stone et al., 1996).

The diagnostic validity of research interviews, such as the CIDI, depends in part on the effectiveness of the screening criteria used to shorten interviews by skipping respondents out of sections. In addition to examining overall diagnostic and discriminant validity, we test the impact of screening criteria on CIDI validity. All CIDI diagnoses are assessed using a general screener, followed by more specific diagnostic items for respondents who endorsed screening questions. Furthermore, we investigate CIDI multi‐informant validity for MDE/DYS diagnoses for which both parents and adolescents provided data. Discrepancies between parent and adolescent reports of depressive symptoms have consistently been noted in the literature (Hope et al., 1999; Jensen et al., 1999a), and are largely attributed to more accurate reporting of internalizing disorders by adolescents than by parents who may not be aware of their child's internal state (Birmaher et al., 1996; Smith, 2007).

Methods

Sample

The NCS‐A is a nationally‐representative face‐to‐face survey of 10,148 adolescents ages 13–17 in the continental United States sampled from household (n = 904) and school (n = 9244) frames (Merikangas et al., 2009). The NCS‐A response rate in the household sample was 85.9% (conditional on adult participation in the National Comorbidity Survey Replication, a nationally representative household survey of adults; Kessler and Merikangas, 2004) and in the school sample was 74.7% (conditional on school participation). Of the 289 schools initially contacted to participate in the NCS‐A, only 81 schools agreed (with refusal to participate most often due to reluctance to release student records for research studies). For each school refusal, a matched replacement school was selected. The final NCS‐A sample included 320 schools rather than the original 289, reflecting the expansion of this recruitment. Comparison of refusal and replacement schools indicated that the use of replacement schools did not introduce any bias into estimates of disorder prevalence and treatment (Kessler et al., 2009a, 2009b). NCS‐A data were weighted for within‐household probability of selection (in the household sub‐sample) and to remove residual discrepancies of socio‐demographic and geographic distributions from corresponding distributions of US residents in the 2000 Census. Details of NCS‐A design and weighting are reported elsewhere (Kessler et al., 2009a, 2009b). The Human Subjects Committees of both Harvard Medical School and the University of Michigan approved recruitment, consent, and field procedures.

The NCS‐A clinical reappraisal study was completed by telephone with a quota sample of 347 adolescent respondents from the school sample and their parents (Kessler et al., 2009c). Adolescents who met DSM‐IV/CIDI criteria for one or more relatively uncommon disorders (e.g. bipolar I or II, agoraphobia) were oversampled relative to respondents who met criteria only for more common disorders or for none, to acquire a large enough sample of adolescents with each disorder to conduct disorder‐specific analyses of CIDI/K‐SADS validity. MDE and DYS validity analyses are based on 321 parent–adolescent dyads with complete data. Respondents received a $50 incentive for each completed CIDI or K‐SADS interview. (For details on the NCS‐A clinical reappraisal study see Kessler et al., 2009c.)

Measures

The CIDI is a fully‐structured diagnostic interview administered by trained lay interviewers to provide clinical diagnoses reflecting DSM‐IV criteria (Merikangas et al., 2009). After warm‐up questions, the CIDI administers screening questions including diagnostic stem questions for GAD, PTSD, and MDE. Positive responses to the screener items are probed in subsequent CIDI sections, which include a series of questions corresponding to DSM‐IV criteria for each of these disorders. DYS is only evaluated in adolescents who do not meet full criteria for MDE, but report a depressed mood for at least one year. As the same items diagnose MDE and DYS, we were unable to develop an independent diagnosis of DYS; instead, MDE and DYS were combined into a single diagnosis. Although the CIDI evaluates 30‐day, 12‐month, and lifetime disorder criteria, we focus here exclusively on lifetime diagnoses.

As described in more detail elsewhere (Kessler et al., 2009a); parents completed a self‐administered questionnaire (SAQ) to provide complementary information about a small number of disorders for which parent report have previously been shown to play a large part in diagnosis: behavior disorders (Grills and Ollendick, 2002; Johnston and Murray, 2003) and depression/dysthymia (Braaten et al., 2001). In the assessment of depression, an SAQ screening question asks parents if their child “ever had episodes of low mood lasting two weeks or longer.” Parents endorsing the screener are directed to complete additional questions about depressive symptoms. If responses do not meet criteria for MDE, but clinically significant symptoms persisted for more than one year, a diagnosis of DYS is assigned. MDE and MDE/DYS diagnoses were generated for parents, adolescents, and a combined diagnosis using an “or” rule at the criterion level.

The CIDI was validated against lifetime diagnoses assessed using the K‐SADS (Kaufman et al., 1997), a semi‐structured clinician‐administered diagnostic interview, administered by telephone to adolescents and one of their parents. Clinicians administering the K‐SADS in this study completed training with an experienced K‐SADS trainer and one of the developers of the K‐SADS and were closely supervised (Kessler et al., 2009c). Clinicians were blinded to CIDI results, with the exception of responses to the screening questions, as prior studies have indicated that respondents in community surveys tend to report less as they are interviewed more often because of response fatigue, leading to the biased perception that earlier interviews over‐estimate prevalence compared to later ones (Bromet et al., 1986; Jensen et al., 1999b). To address this problem, clinical interviewers were informed of responses to diagnostic stem questions, but not whether respondents met full criteria for disorders. Because the vast majority of respondents who endorse CIDI stem questions do not go on to meet full DSM‐IV/CIDI criteria for the associated disorder, this partial un‐blinding of interviewers did not inform clinical interviewers whether the CIDI diagnosis was positive. All disorders in both the CIDI and K‐SADS were diagnosed using DSM‐IV organic exclusions and diagnostic hierarchy rules. Final diagnoses, which are considered here, combined information from adolescents and parents.

Analysis methods

We weighted the full clinical reappraisal sample (n = 347) to adjust for the over‐sampling of CIDI cases and post‐stratified for small residual discrepancies between the weighted clinical reappraisal sample and the full weighted NCS‐A sample on a wide range of variables. Prevalence estimates were compared using McNemar χ2 tests that take into account unequal sampling weights. Individual‐level concordance was evaluated treating the composite K‐SADS diagnoses as the gold standard by calculating CIDI sensitivity (SN), specificity (SP), positive predictive value (PPV), negative predictive value (NPV), Cohen's ĸ (Cohen, 1960), and the area under the receiver operating characteristic curve (AUC) of a CIDI diagnosis predicting the K‐SADS diagnosis.

To assess discriminant validity, we first conducted a series of binary logistic regressions predicting each of the three K‐SADS distress diagnoses from all three CIDI diagnoses, to determine the extent to which CIDI disorders predicted corresponding diagnoses on the K‐SADS versus other distress disorders. If the CIDI is able to effectively discriminate between disorders, we would expect each CIDI diagnosis to most strongly predict the same diagnosis on the K‐SADS, despite disorder comorbidity. We next examined cases that had one distress diagnosis but not another (e.g. PTSD but not GAD) in both the K‐SADS and the CIDI. For each pair of disorders we assessed agreement between the K‐SADS and the CIDI using Chi‐square (χ 2) tests and Cohen's κ. This allowed us to determine whether the CIDI was accurately able to distinguish among distress disorders among adolescents. All analyses were completed using SAS 9.0 and SUDAAN 9.0.1 software (Research Triangle Institute, 2005; SAS Institute Inc., 2002).

Results

CIDI concordance with K‐SADS for GAD

CIDI GAD had very good concordance with K‐SADS diagnoses (κ = 0.65, AUC = 0.78) and provided a comparable estimate of prevalence (CIDI percent = 2.4, K‐SADS percent = 3.3, χ 2 = 1.21, p = 0.28) (Table 1). CIDI rates for Criteria B (trouble controlling worry), C (one or more physical symptom), D (focus not other Axis I disorder), and E (impairment) were higher than K‐SADS estimates (ratios from 2.0 to 4.5) but were offset by higher K‐SADS reports of Criterion A, worrying frequently and excessively for a duration of six months (ratio of K‐SADS to CIDI = 2.4), part of the GAD screener. Removing Criterion F (disturbance not due to substance or general medical condition) from the diagnostic criteria slightly improved K‐SADS diagnostic prediction (AUC = 0.80), as reported in Kessler et al. (2009c); however, further analysis revealed that Criterion F improved PPV and we decided to include it in the algorithm for future analyses. We also examined the CIDI screening criteria for GAD and found that the initial screener eliminated 31% of respondents, none of whom received a K‐SADS diagnosis. Similarly, very few adolescents excluded by subsequent GAD screeners (0% to 5% for each) received a K‐SADS diagnosis.

Table 1.

Concordance of CIDI/DSM‐IV GAD symptoms with K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 347)

Prevalence Concordancea
K‐SADS CIDI SN SP PPV NPV Kappa AUC
Percent (SE) Percent (SE) χ 2 Est (SE) Est (SE) Est (SE) Est (SE) Est
Criterion A: Excessive worry 9.0 (1.7) 3.7 (0.9) 13.3* 25.2 (7.4) 98.4 (0.7) 61.2 (12.8) 93.0 (1.6) 0.32 0.62
Criterion B: Trouble controlling worry 8.6 (1.6) 28.1 (2.7) 53.0* 63.1 (9.8) 75.2 (2.8) 19.3 (4.0) 95.6 (1.5) 0.19 0.69
Criterion C: Physical symptoms 8.9 (1.6) 17.7 (2.2) 16.8* 53.2 (9.2) 85.8 (2.2) 26.9 (5.6) 94.9 (1.4) 0.27 0.70
Criterion D: Focus not other Axis I disorder 13.2 (2.0) 59.7 (3.2) 145.7* 84.1 (6.7) 44.0 (3.4) 18.6 (2.9) 94.8 (2.4) 0.11 0.64
Criterion E: Impairment 10.6 (1.8) 25.2 (2.5) 28.5* 60.4 (9.1) 78.9 (2.6) 25.3 (4.5) 94.4 (1.8) 0.24 0.70
Criterion F: Not due to substance 14.2 (2.1) 13.1 (2.0) 0.2 29.5 (6.5) 89.6 (2.0) 32.1 (6.9) 88.5 (2.2) 0.20 0.60
Overall diagnosis: (A, B, C, D, E, F) 3.3 (1.0) 2.4 (0.8) 1.2 57.2 (13.9) 99.5 (0.3) 78.8 (12.6) 98.6 (0.6) 0.65 0.78
Open in a new tab
a

SN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; Kappa, Cohen's Kappa; AUC, area under the receiver operating characteristic curve.

*

The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test.

CIDI concordance with K‐SADS for PTSD

CIDI PTSD diagnoses had fairly good concordance with the K‐SADS (κ = 0.47, AUC = 0.72) and comparable prevalence (CIDI percent = 3.8, K‐SADS percent = 4.2, χ 2 = 0.15, p = 0.70), but low sensitivity (SN = 46.8%) (Table 2). Because of this low sensitivity, several strategies were tested to relax the CIDI PTSD diagnostic criteria, allowing more flexibility in symptom requirements. The revised diagnostic algorithm with the best validity required respondents to either meet all six DSM‐IV criteria (A–F) or, if they met A and E (traumatic event/fear response and one month duration), but failed to meet one of the other DSM‐IV criteria, required them to have a combined total of nine symptoms distributed in any way across Criteria B, C, and D (rather than the minimum of six required by the DSM‐IV). With these new criteria, those with a high number of symptoms had a second chance to be assigned a PTSD diagnosis. Using this modification, prevalence estimates remained close (CIDI percent = 4.4, K‐SADS percent = 4.2, χ 2 = 0.05, p = 0.83) and K‐SADS concordance improved (SN = 59.9%, κ = 0.56, AUC = 0.79). PTSD screeners were effective: the initial Criterion A1 screener excluded 31% of adolescents, less than 1% of whom received a K‐SADS diagnosis. Two other early screening items about traumatic reactions and their duration excluded 50% of adolescents, only 2% of whom had a K‐SADS diagnosis.

Table 2.

Concordance of CIDI/DSM‐IV PTSD symptoms with K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 347)

Prevalence Concordancea
K‐SADS CIDI SN SP PPV NPV Kappa AUC
Percent (SE) Percent (SE) χ 2 Est (SE) Est (SE) Est (SE) Est (se) Est
Criterion A: Traumatic event and fear response 12.1 (1.9) 10.9 (1.7) 0.4 43.3 (7.8) 93.6 (1.4) 48.1 (8.0) 92.3 (1.7) 0.39 0.68
Criterion B: Re‐experiencing 11.1 (1.8) 8.1 (1.5) 2.2 32.5 (7.4) 94.9 (1.3) 44.4 (9.3) 91.9 (1.6) 0.31 0.64
Criterion C: Avoidance/numbing 5.4 (1.2) 6.2 (1.2) 0.3 43.0 (10.1) 95.9 (1.2) 37.6 (8.8) 96.7 (1.1) 0.36 0.69
Criterion D: Hyperarousal 6.2 (1.2) 8.0 (1.5) 1.0 45.9 (9.8) 94.5 (1.4) 35.3 (8.3) 96.4 (1.1) 0.35 0.70
Criterion E: Duration > one month 10.9 (1.7) 7.1 (1.3) 4.5* 34.5 (7.6) 96.3 (1.0) 52.9 (9.6) 92.3 (1.6) 0.36 0.65
Criterion F: Impairment 8.8 (1.5) 6.8 (1.4) 1.1 31.0 (7.2) 95.5 (1.3) 40.2 (9.4) 93.5 (1.4) 0.30 0.63
Overall diagnosis: (A, B, C, D, E, F) 4.2 (1.2) 3.8 (1.0) 0.2 46.8 (14.2) 98.1 (0.8) 51.8 (14.1) 97.7 (1.0) 0.47 0.72
Modified diagnosis b 4.2 (1.2) 4.4 (1.0) 0.0 59.9 (14.2) 98.0 (0.8) 57.0 (12.7) 98.2 (0.9) 0.56 0.79
Open in a new tab
a

SN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; Kappa, Cohen's Kappa; AUC, area under the receiver operating characteristic curve.

b

The modified PTSD diagnosis requires respondents to either meet all six DSM‐IV criteria (A–F) or, if they met A and E (traumatic event/fear response and one month duration), but failed to meet one of the other DSM‐IV criteria, they were required to have a combined total of nine symptoms distributed in any way across Criteria B, C, and D (rather than the minimum of six required by the DSM‐IV).

*

The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test.

CIDI concordance with K‐SADS for depressive disorders

Adolescent‐only and parent‐only CIDI reports of MDE both underestimated disorder prevalence (12.6% and 8.6% respectively, compared with the 17.5% K‐SADS prevalence; χ 2 = 6.86, 25.10, p < 0.05 for both) (Part I of Table 3). However, the composite (parent and adolescent combined) report closely approximated K‐SADS prevalence (CIDI percent = 17.7, K‐SADS percent = 17.5, χ 2 = 0.01, p = 0.94) and correctly identified most adolescents with the disorder (SN = 78.4%). For the combined MDE/DYS diagnosis, the composite report was again a stronger estimate of K‐SADS diagnosis (CIDI percent = 18.0, K‐SADS percent = 19.8, χ 2 = 1.14, p = 0.29; SN = 76.5%) (Part II of Table 3). For individual criteria, parent report almost always had better concordance with the K‐SADS (AUC range from 0.59 to 0.72) than youth report (AUC range from 0.57 to 0.68), but for every criterion the composite CIDI had the best agreement with the K‐SADS, due to improved sensitivity (Table 4). Adolescents reported higher rates of CIDI symptoms than their parents for seven out of nine symptoms (adolescent symptom prevalences ranged from 5.0% to 17.0%, parent symptom prevalence ranged from 5.6% to 11.6%). The greatest discrepancy was for the question assessing thoughts of death, endorsed by 11.0% of adolescents, but only 4.2% of parents (ratio = 2.6).

Table 3.

Diagnostic concordance of CIDI/DSM‐IV MDE and MDE/DYS diagnoses with composite K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 321)

Prevalence Concordancea
K‐SADS CIDI SN SP PPV NPV Kappa AUC
Percent (SE) Percent (SE) χ 2 Est (SE) Est (SE) Est (SE) Est (SE) Est
Part I: Major Depressive Episode
MDE Youth CIDI versus Clinician K‐SADS 17.5 (2.4) 12.6 (2.0) 6.9* 53.9 (7.2) 96.2 (1.3) 75.0 (7.3) 90.7 (2.0) 0.56 0.75
MDE Parent CIDI versus Clinician K‐SADS 17.5 (2.4) 8.6 (1.6) 25.1* 43.1 (7.3) 98.8 (1.1) 88.1 (9.9) 89.1 (2.1) 0.52 0.71
MDE Composite CIDI versus Clinician K‐SADS 17.5 (2.4) 17.7 (2.1) 0.0 78.4 (7.3) 95.3 (1.6) 77.8 (7.0) 95.4 (1.8) 0.74 0.87
Part II: Major Depressive Episode or Dysthymic Disorder
MDE/DYS Youth CIDI versus Clinician K‐SADS 19.8 (2.5) 13.0 (2.1) 10.5* 52.4 (6.8) 96.8 (1.3) 80.1 (6.9) 89.1 (2.2) 0.57 0.75
MDE/DYS Parent CIDI versus Clinician K‐SADS 19.8 (2.5) 9.1 (1.6) 33.9* 43.7 (6.7) 99.5 (0.5) 95.6 (4.2) 87.7 (2.2) 0.54 0.72
MDE/DYS Composite CIDI versus K‐SADS 19.8 (2.5) 18.0 (2.2) 1.1 76.5 (6.4) 96.4 (1.4) 84.2 (5.7) 94.3 (1.9) 0.75 0.86
Open in a new tab
a

SN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; Kappa, Cohen's Kappa; AUC, area under the receiver operating characteristic curve.

*

The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test.

Table 4.

Concordance of CIDI/DSM‐IV MDE Criterion A2 symptoms with K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 321)

Prevalence Concordancea
K‐SADS Composite CIDI SN SP PPV NPV Kappa AUC
Percent (SE) Percent (SE) χ 2 Est (SE) Est (SE) Est (SE) Est (SE) Est
Depressed 24.1 (2.6) 23.1 (2.5) 2.8 49.6 (6.1) 85.4 (2.4) 51.9 (5.9) 84.2 (2.7) 0.36 0.68
Loss of interest 15.6 (2.2) 19.3 (2.3) 42.9* 51.9 (7.5) 86.8 (2.1) 42.0 (6.3) 90.7 (2.0) 0.35 0.69
Weight change 9.4 (1.7) 17.0 (2.2) 195.4* 45.3 (9.4) 85.9 (2.1) 24.9 (5.5) 93.8 (1.7) 0.23 0.66
Sleep problems 14.5 (2.1) 19.6 (2.3) 94.1* 61.4 (7.5) 87.5 (2.1) 45.5 (6.2) 93.0 (1.8) 0.43 0.74
Psychomotor disturbance 7.9 (1.7) 14.0 (2.0) 132.2* 33.3 (10.0) 87.7 (2.0) 18.8 (5.9) 93.9 (1.7) 0.16 0.60
Fatigue 14.2 (2.0) 20.0 (2.4) 132.7* 67.3 (7.1) 87.8 (2.1) 47.7 (6.4) 94.2 (1.5) 0.47 0.78
Worthless 7.7 (1.5) 11.5 (1.8) 84.8* 58.7 (10.3) 92.4 (1.6) 39.3 (7.8) 96.4 (1.2) 0.42 0.76
Trouble concentrating 12.4 (1.9) 21.2 (2.4) 224.4* 53.8 (7.9) 83.5 (2.4) 31.6 (5.4) 94.7 (1.8) 0.29 0.69
Thoughts of death 5.6 (1.3) 12.7 (1.8) 242.7* 54.9 (12.2) 89.9 (1.7) 24.4 (6.2) 97.1 (1.1) 0.28 0.72
Open in a new tab
a

SN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; Kappa, Cohen's Kappa; AUC, area under the receiver operating characteristic curve.

*

The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test.

Of the 35.9% of adolescents who denied MDE screener questions, only 6.1% later obtained a K‐SADS diagnosis, suggesting that the adolescent screener was effective. However, of the 88.4% of parents who denied the SAQ MDE screener question, 12.0% of their children went on to receive a K‐SADS MDE/DYS diagnosis but only 9.5% received a CIDI MDE/DYS diagnosis (due to the absence of any parental contribution to symptom reports). In comparison, 79.4% of adolescents whose parents endorsed the screener obtained a K‐SADS diagnosis. Thus, although the parental screener was highly specific, it also reduced the sensitivity of the CIDI for those K‐SADS cases that were not identified by the screener (about 53.4% of all K‐SADS‐identified cases).

CIDI distress disorder discriminant validity

We next examined the discriminant validity of CIDI diagnoses of distress disorders. In logistic regressions, CIDI diagnosis of each disorder is a much stronger predictor of the corresponding K‐SADS disorder [odds ratio (OR) = 59.03–91.47, all significant at p < 0.001], than of any of the other distress disorders (OR = 1.12–3.85) (Table 5). As a particularly stringent test of discriminant validity, we used the sample of respondents who met criteria for any of the three disorders based on both the CIDI and K‐SADS (n = 99) and for each pair of diagnoses we examined the cases that had exactly one of the two diagnoses in both the CIDI and the K‐SADS. For each such pair, the CIDI and K‐SADS diagnoses were significantly associated (p < 0.001) and associations were strong for discriminating between GAD and MDE/DYS [κ = 0.86, standard error (SE) = 0.17], between PTSD and MDE/DYS (κ = 0.80, SE = 0.16), and between GAD and PTSD (κ = 0.96, SE = 0.08).

Table 5.

Odds ratios (OR) of binary logistic regressions of K‐SADS diagnoses on CIDI distress disorder diagnoses in the NCS‐A clinical reappraisal sample (n = 321)

K‐SADS outcomes
CIDI Predictors MDE/DYS GAD PTSD
OR (95% CI) OR (95% CI) OR (95% CI)
MDE/DYS 59.03* (23.31–149.46) 3.85 (0.60–24.73) 1.12 (0.32–3.94)
GAD 2.23 (0.48–10.35) 91.47* (17.99–464.05) 1.33 (0.14–12.68)
PTSD 1.89 (0.22–16.12) 1.20 (0.08–17.46) 69.13* (14.44–330.30)
Open in a new tab
*

p < 0.001.

Discussion

Results support the concurrent validity of individual CIDI diagnoses of GAD and MDE/DYS with K‐SADS diagnoses. Although the original CIDI PTSD diagnosis had low sensitivity, concordance with the K‐SADS was improved when we allowed for more variability in symptom endorsement. As discussed elsewhere (Alegria et al., 2009), the structured CIDI interview does not allow for the same clinical interpretation of traumatic events and related symptoms that are elicited by the broader K‐SADS questions. By modifying the PTSD criteria, we introduced flexibility in CIDI diagnostic assignment that allowed for a better estimate of clinical diagnosis. With this modification, the CIDI and K‐SADS had good concordance for each of the three diagnoses of distress disorders. Concordance of individual symptoms was also strong, although somewhat weaker than overall diagnostic validity. If the diagnosis represents a single underlying dimension that is assessed by individual symptoms, then combining symptoms would be expected to improve reliability over separate measurement of criteria.

For MDE/DYS, parent‐only reports significantly under‐diagnosed depression, compared to adolescent‐only and clinician diagnostic assignments. This finding suggests that at least some parents have very limited knowledge of their adolescents' depressive symptoms and confirms the importance of a multi‐informant approach to MDE/DYS assessment (Birmaher et al., 1996; Smith, 2007). In particular, the criterion “thoughts of death” was endorsed by almost three times as many adolescents as parents, indicating that, consistent with prior research, many parents may be unaware of suicidal ideation among their children (Breton et al., 2002; Kashani et al., 1989; Sourander et al., 1999).

Given the role of screening questions in determining CIDI diagnostic‐assignment, we were interested in whether screener criteria were effective in skipping out respondents least likely to meet diagnostic criteria on the K‐SADS. GAD, PTSD, and MDE/DYS screeners in the adolescent interview effectively screen out a large number (31–83%) of respondents who rarely have a K‐SADS diagnosis (0–6%). However, the performance of the screening item for depression at the beginning of the parent SAQ is only moderate. With 12% of those skipped out of the SAQ section later meeting K‐SADS criteria for MDE/DYS, this screener prevents a substantial number (53.4%) of parents of true cases from responding to diagnostic questions.

Our discriminant validity analysis suggests that among those respondents who meet criteria for any of the three distress disorders, the CIDI is able to effectively distinguish among GAD, PTSD, and MDE/DYS diagnoses. Despite comorbidity among distress disorders, each of the CIDI diagnoses is much more strongly associated with the corresponding K‐SADS diagnoses than with any of the other distress disorders. Further, among adolescents who have one K‐SADS distress disorder but not another, the CIDI effectively discriminates among pairs of disorders. Prior validity studies have not typically assessed this type of joint validity, which can have considerable implications for studying comorbidity among disorders, such as distress disorders, that are distinct, but highly overlapping (Clark and Watson, 2006; Krueger, 1999; Watson, 2005).

This study has several limitations. First, the CIDI was administered in‐person and the K‐SADS was administered over the phone. This discrepancy potentially introduces an important method effect that could have caused underestimation of the concordance of these measures. However, telephone administration of clinical interviews is now widely used and accepted based on evidence of comparable validity to in‐person interviews in clinical reappraisal studies (Aneshensel et al., 1982; Sobin et al., 1993; Rohde et al., 1997). Telephone administration has the advantage of centralized and closely supervised clinical interview staff who do not have the geographic restrictions of face‐to‐face clinical interviews. Here it provided the only feasible option for conducting a large NCS‐A clinical reappraisal study. Further, changes in mental state over the elapsed time between CIDI and K‐SADS administration could have caused us to underestimate concordance. Finally, the parent‐report MDE section screened out parents who reported that their child did not have symptoms of depression; however, our analysis of screener data indicated that a substantial proportion of parents who skipped out at this point had children who later received a K‐SADS diagnosis.

Given these findings, we have two primary suggestions for improving the CIDI for future use with adolescents. First, we recommend relaxing CIDI requirements for PTSD to no longer require endorsement of at least one symptom in each cluster. Allowing for more flexibility in symptom endorsement, as we did here, improved sensitivity, thereby preventing CIDI under‐identification of youth with PTSD. Second, we recommend modifying the depression screening item at the beginning of the parent SAQ, as this question skipped out a substantial number of parents whose children later went on to receive a K‐SADS diagnosis of MDE/DYS. Altering the screener to be more sensitive (perhaps by removing the duration requirement and/or adding language that normalizes episodes of depressive symptoms in adolescents), may improve parent assessment of MDE/DYS, although further testing would be required. With these modifications, the CIDI appears to be an effective tool for diagnosing and differentiating among distress disorder.

Declaration of interest statement

Dr Kessler has been a consultant for AstraZeneca, Analysis Group, Bristol‐Myers Squibb, Cerner‐Galt Associates, Eli Lilly & Company, GlaxoSmithKline Inc., HealthCore Inc., Health Dialog, Integrated Benefits Institute, John Snow Inc., Kaiser Permanente, Matria Inc., Mensante, Merck & Co, Inc., Ortho‐McNeil Janssen Scientific Affairs, Pfizer Inc., Primary Care Network, Research Triangle Institute, Sanofi‐Aventis Groupe, Shire US Inc., SRA International, Inc., Takeda Global Research & Development, Transcept Pharmaceuticals Inc., and Wyeth‐Ayerst; has served on advisory boards for Appliance Computing II, Eli Lilly & Company, Mindsite, Ortho‐McNeil Janssen Scientific Affairs, Plus One Health Management and Wyeth‐Ayerst; and has had research support for his epidemiological studies from Analysis Group Inc., Bristol‐Myers Squibb, Eli Lilly & Company, EPI‐Q, GlaxoSmithKline, Johnson & Johnson Pharmaceuticals, Ortho‐McNeil Janssen Scientific Affairs., Pfizer Inc., Sanofi‐Aventis Groupe, and Shire US, Inc. The remaining authors have no competing interests.

Acknowledgments

The National Comorbidity Survey Replication Adolescent Supplement (NCS‐A) is supported by the National Institute of Mental Health (NIMH; U01‐MH60220 and R01‐MH66627) with supplemental support from the National Institute on Drug Abuse (NIDA), the Substance Abuse and Mental Health Services Administration (SAMHSA), the Robert Wood Johnson Foundation (RWJF; Grant 044780), and the John W. Alden Trust. The views and opinions expressed in this report are those of the authors and should not be construed to represent the views of any of the sponsoring organizations, agencies, or US Government. A complete list of NCS‐A publications can be found at http://www.hcp.med.harvard.edu/ncs. Send correspondence to ncs@hcp.med.harvard.edu. The NCS‐A is carried out in conjunction with the World Health Organization World Mental Health (WMH) Survey Initiative. We thank the staff of the WMH Data Collection and Data Analysis Coordination Centers for assistance with instrumentation, fieldwork, and consultation on data analysis. The WMH Data Coordination Centers have received support from NIMH (R01‐MH070884, R13‐MH066849, R01‐MH069864, R01‐MH077883), NIDA (R01‐DA016558), the Fogarty International Center of the National Institutes of Health (FIRCA R03‐TW006481), the John D. and Catherine T. MacArthur Foundation, the Pfizer Foundation, and the Pan American Health Organization. The WMH Data Coordination Centers have also received unrestricted educational grants from Astra Zeneca, BristolMyersSquibb, Eli Lilly and Company, GlaxoSmithKline, Ortho‐McNeil, Pfizer, Sanofi‐Aventis, and Wyeth. A complete list of WMH publications can be found at http://www.hcp.med.harvard.edu/wmh/.

References

  1. Alegria M., Shrout P.E., Torres M., Lewis‐Fernández R., Abelson J.M., Powell M., Interian A., Lin J., Laderman M., Canino G. (2009) Lessons learned from the clinical reappraisal study of the Composite International Diagnostic Interview with Latinos. International Journal of Methods in Psychiatric Research, 18, 84–95, DOI: 10.1002/mpr.280 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aneshensel C.S., Frerichs R.R., Clark V.A., Yokopenic P.A. (1982) Measuring depression in the community: a comparison of telephone and personal interviews. Public Opinion Quarterly, 46, 110–121. [DOI] [PubMed] [Google Scholar]
  3. Birmaher B., Ryan N.D., Williamson D.E., Brent D.A., Kaufman J. (1996) Childhood and adolescent depression: a review of the past 10 years Part II. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 1575–1583, DOI: 10.1097/00004583-199612000-00008 [DOI] [PubMed] [Google Scholar]
  4. Braaten E.B., Biederman J., DiMauro A., Mick E., Monuteaux M.C., Muehl K., Faraone S.V. (2001) Methodological complexities in the diagnosis of major depression in youth: an analysis of mother and youth self‐reports. Journal of Child and Adolescent Psychopharmacology, 11, 395–407, DOI: 10.1089/104454601317261573 [DOI] [PubMed] [Google Scholar]
  5. Breton J.J., Tousignant M., Bergeron L., Berthiaume C. (2002) Informant‐specific correlates of suicidal behavior in a community survey of 12‐ to 14‐year‐olds. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 723–730, DOI: 10.1097/00004583-200206000-00012 [DOI] [PubMed] [Google Scholar]
  6. Bromet E.J., Dunn L.O., Connell M.M., Dew M.A., Schulberg H.C. (1986) Longterm reliability of diagnosing lifetime major depression in a community sample. Archives of General Psychiatry, 43, 435–440. [DOI] [PubMed] [Google Scholar]
  7. Brooks S.J., Kutcher S. (2001) Diagnosis and measurement of adolescent depression: a review of commonly utilized instruments. Journal of Child and Adolescent Psychopharmacology, 11, 341–376, DOI: 10.1089/104454601317261546 [DOI] [PubMed] [Google Scholar]
  8. Clark L.A., Watson D. (2006) Distress and fear disorders: an alternative empirically based taxonomy of the “mood” and “anxiety” disorders. The British Journal of Psychiatry, 189, 481–483, DOI: 10.1192/bjp.bp.106.03825 [DOI] [PubMed] [Google Scholar]
  9. Cohen J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46, DOI: 10.1177/001316446002000104 [DOI] [Google Scholar]
  10. Ezpeleta L., de la Osa N., Doménech J.M., Navarro J.B., Losilla J.M., Júdez J. (1997) Diagnostic agreement between clinicians and the Diagnostic Interview for Children and Adolescents – DICA‐R – in an outpatient sample. Journal of Child Psychology and Psychiatry, 38, 431–440, DOI: 10.1111/j.1469-7610.1997.tb01528.x [DOI] [PubMed] [Google Scholar]
  11. Green J.G., Avenevoli S., Finkelman M., Gruber M.J., Kessler R.C., Merikangas K.R., Sampson N.A., Zaslavsky A.M. (2011) Validation of the diagnoses of panic disorder and phobic disorders in the US National Comorbidity Survey Replication Adolescent (NCS‐A) Supplement. International Journal of Methods in Psychiatric Research, 20(2), 105–115, DOI: 10.1002/mpr.336 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Green J.G., Finkelman M., Gruber M., Kessler R.C., Sampson N.A., Zaslavsky A.M. (2010) Attention deficit/hyperactivity disorder: concordance of the adolescent version of the Composite International Diagnostic Interview Version 3.0 (CIDI‐A) with the K‐SADS in the US National Comorbidity Survey Adolescent (NCS‐A) Supplement. International Journal of Methods in Psychiatric Research, 19, 34–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Grills A.E., Ollendick T.H. (2002) Issues in parent–child agreement: the case of structured diagnostic interviews. Clinical Child and Family Psychology Review, 5, 57–83, DOI: 10.1023/A:1014573708569 [DOI] [PubMed] [Google Scholar]
  14. Hope T.L., Adams C., Reynolds L., Powers D., Perez R.A., Kelley M.L. (1999) Parent vs. self‐report: contributions toward diagnosis of adolescent psychopathology. Journal of Psychopathology and Behavioral Assessment, 21, 349–363, DOI: 10.1023/A:1022124900328 [DOI] [Google Scholar]
  15. Jensen P.S., Rubio‐Stipec M., Canino G., Bird H.R., Dulcan M.K., Schwab‐Stone M.E., Lahey B.B. (1999a) Parent and child contributions to diagnosis of mental disorder: are both informants always necessary? Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1569–1579, DOI: 10.1097/00004583-199912000-00019 [DOI] [PubMed] [Google Scholar]
  16. Jensen P.S., Watanabe H.K., Richters J.E. (1999b) Who's up first? Testing for order effects in structured interviews using a counterbalanced experimental design. Journal of Abnormal Child Psychology, 27, 439–445, DOI: 10.1023/A:1021927909027 [DOI] [PubMed] [Google Scholar]
  17. Johnston C., Murray C. (2003) Incremental validity in the psychological assessment of children and adolescents. Psychology Assessment, 15, 496–507. [DOI] [PubMed] [Google Scholar]
  18. Kashani J.H., Goddard P., Reid J.C. (1989) Correlates of suicidal ideation in a community sample of children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry, 28, 912–917, DOI: 10.1097/00004583-198911000-00016 [DOI] [PubMed] [Google Scholar]
  19. Kaufman J., Birmaher B., Brent D., Rao U., Flynn C., Moreci P., Williamson D., Ryan N. (1997) Schedule for Affective Disorders and Schizophrenia for School‐age Children – Present and Lifetime version (K‐SADS‐PL): initial reliability and validity data. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 980–988, DOI: 10.1097/00004583-199707000-00021 [DOI] [PubMed] [Google Scholar]
  20. Kessler R.C., Avenevoli S., Costello E.J., Green J.G., Gruber M.J., Heeringa S., Merikangas K.R., Pennell B.E., Sampson N.A., Zaslavsky A.M. (2009a) Design and field procedures in the US National Comorbidity Survey Replication Adolescent Supplement (NCS‐A). International Journal of Methods in Psychiatric Research, 18, 69–83, DOI: 10.1002/mpr.279 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kessler R.C., Avenevoli S., Costello E.J., Green J.G., Gruber M.J., Heeringa S., Merikangas K.R., Pennell B.E., Sampson N.A., Zaslavsky A.M. (2009b) The National Comorbidity Survey Adolescent Supplement (NCS‐A): II. Overview and design. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 380–385, DOI: 10.1097/CHI.0b013e3181999705 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kessler R.C., Cox B.J., Green J.G., Ormel J., McLaughlin K.A., Merikangas K.M., Petukhova M., Pine D.S., Russo L.J., Swendson J., Wittchen H.‐U., Zaslavksky A.M. (2011) The effects of latent variables in the development of comorbidity among common mental disorders. Depression and Anxiety, 28, 29–39, DOI: 10.1002/da.20760 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kessler R.C., Avenevoli S., Green J.G., Gruber M.J., Guyer M., He Y., Jin R., Kaufman J., Sampson N.A., Zaslavsky A.M., Merikangas K.R. (2009c) The National Comorbidity Survey Adolescent Supplement (NCS‐A): III. Concordance of DSM‐IV/CIDI diagnoses with clinical reassessments. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 386–399, DOI: 10.1097/CHI.0b013e31819a1cbc [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kessler R.C., Merikangas K.R. (2004) The National Comorbidity Survey Replication (NCS‐R): background and aims. International Journal of Methods in Psychiatric Research, 13, 60–68, DOI:  10.1002/mpr.166 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kessler R.C., Üstün T.B. (2004) The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). International Journal of Methods in Psychiatric Research, 13, 93–121, DOI: 10.1002/mpr.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Krueger R.F. (1999) The structure of common mental disorders. Archives of General Psychiatry, 56, 921–926. [DOI] [PubMed] [Google Scholar]
  27. Merikangas K.R., Avenevoli S., Costello E.J., Koretz D., Kessler R.C. (2009) The National Comorbidity Survey Replication Adolescent Supplement (NCS‐A): I Background and measures. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 367–369, DOI: 10.1097/CHI.0b013e31819996f1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Research Triangle Institute . (2005) SUDAAN: Professional Software for Survey Data Analysis, version 9.0.1, Research Triangle Park, NC, Research Triangle Institute.
  29. Rohde P., Lewinsohn P.M., Seeley J.R. (1997) Comparability of telephone and face‐to‐face interviews in assessing axis I and II disorders. The American Journal of Psychiatry, 154, 1593–1598. [DOI] [PubMed] [Google Scholar]
  30. SAS Institute Inc . (2002) SAS 9.1.3 Help and Documentation, Cary, NC, SAS Publishing.
  31. Schwab‐Stone M.E., Shaffer D., Dulcan M.K., Jensen P.S., Fisher P., Bird H.R., Goodman S.H., Lahey B.B., Lichtman J.H., Canino G., Rubio‐Stipec M., Rae D.S. (1996) Criterion validity of the NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC‐2.3). Journal of the American Academy of Child and Adolescent Psychiatry, 35, 878–888, DOI: 10.1097/00004583-199607000-00013 [DOI] [PubMed] [Google Scholar]
  32. Smith S.R. (2007) Making sense of multiple informants in child and adolescent psychopathology: a guide for clinicians. Journal of Psychoeducational Assessment, 25, 139–149, DOI: 10.1177/0734282906296233 [DOI] [Google Scholar]
  33. Sobin E., Weissman M.M., Goldstein R.B., Adams P. (1993) Diagnostic interviewing for family studies: comparing telephone and face‐to‐face methods for the diagnosis of lifetime psychiatric disorders. Psychiatric Genetics, 3, 227–233. [Google Scholar]
  34. Sourander A., Helstela L., Helenius H. (1999) Parent–adolescent agreement on emotional and behavioral problems. Social Psychiatry and Psychiatric Epidemiology, 34, 657–663, DOI: 10.1007/s001270050189 [DOI] [PubMed] [Google Scholar]
  35. Watson D. (2005) Rethinking the mood and anxiety disorders: a quantitative hierarchical model for DSM‐V. Journal of Abnormal Psychology, 114, 522–536, DOI: 10.1037/0021-843X.114.4.522 [DOI] [PubMed] [Google Scholar]

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES