Validation of the diagnoses of panic disorder and phobic disorders in the US National Comorbidity Survey Replication Adolescent (NCS‐A) supplement

Abstract

Validity of the adolescent version of the World Health Organization Composite International Diagnostic Interview (CIDI) Version 3.0, a fully‐structured research diagnostic interview designed to be used by trained lay interviewers, is assessed in comparison to independent clinical diagnoses based on the Schedule for Affective Disorders and Schizophrenia for School‐age Children (K‐SADS). This assessment is carried out in the clinical reappraisal sub‐sample (n = 347) of the US National Comorbidity Survey Adolescent (NCS‐A) supplement, a large (n = 10,148) community epidemiological survey of the prevalence and correlates of adolescent mental disorders in the United States. The diagnoses considered are panic disorder and phobic disorders (social phobia, specific phobia, agoraphobia). CIDI diagnoses are found to have good concordance with K‐SADS diagnoses [area under the receiver operating characteristic curve (AUC) = 0.81–0.94], although the CIDI diagnoses are consistency somewhat higher than the K‐SADS diagnoses. Data are also presented on criterion‐level concordance in an effort to pinpoint CIDI question series that might be improved in future modifications of the instrument. Finally, data are presented on the factor structure of the fears associated with social phobia, the only disorder in this series where substantial controversy exists about disorder subtypes. Copyright © 2011 John Wiley & Sons, Ltd.

Introduction

While a number of scales and diagnostic interviews have been developed to assess mental disorders among adolescents, only limited evidence exists regarding their validity (Brooks and Kutcher, 2003). This report examines the validity of four modules of a newly developed diagnostic instrument, the adolescent version of the World Health Organization Composite International Diagnostic Interview Version 3.0 (CIDI 3.0) (Merikangas et al., 2009), based on data collected in the US National Comorbidity Survey Adolescent (NCS‐A; Merikangas et al., 2009) supplement. The CIDI is a fully‐structured research diagnostic interview designed to be used by trained lay interviewers and to generate diagnoses according to the definitions and criteria of both the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV) and the International Classification of Diseases (ICD‐10) systems (Kessler and Üstün, 2004). The NCS‐A is a large (n = 10,148) community epidemiological survey of the prevalence and correlates of adolescent mental disorders in the United States. The validation was carried out by comparing diagnoses based on the CIDI with independent clinical diagnoses based on the Schedule for Affective Disorders and Schizophrenia for School‐age Children (K‐SADS; Kaufman et al., 1997). Disaggregated data are also presented on criterion‐level concordance of CIDI and K‐SADS assessments in an effort to pinpoint CIDI question series that might be improved in future modifications of the instrument. The diagnoses considered here are panic disorder and phobic disorders (social phobia, specific phobia, agoraphobia). This paper is part of a series of papers on CIDI diagnostic validity in the NCS‐A (Kessler et al., 2009c; Green et al., in press; Green et al., 2010).

In addition to examining concordance of diagnoses and criteria based on the CIDI and K‐SADS, we also examine the internal structure of the CIDI diagnoses of social phobia, the one diagnosis out of the four considered here where substantial controversy exists about subtypes. Although the DSM‐IV requires only one social fear to assign a diagnosis of social phobia, thus assuming that each social fear contributes equally to the diagnosis, an ongoing debate exists concerning the possibility that some social fears are more important than others (Perugi et al., 2001; Stein and Deutsch, 2003; Vriends et al., 2007). This possibility is suggested by the results of factor analyses of social phobia fears, which have sometimes documented a multidimensional structure that distinguishes subtypes of “interpersonal anxiety” and “performance anxiety” (Kessler et al., 1998; Perugi et al., 2001). An alternative view is that social phobia should be conceptualized as a continuum, with increased number of fears corresponding to increased impairment rather than specifying subtypes based on the content of fears (Ruscio et al., 2008; Vriends et al., 2007). We investigate this issue by studying the factor structure of social phobia responses in the CIDI.

Methods

Samples

The NCS‐A is a nationally‐representative face‐to‐face survey of 10,148 adolescents ages 13–17 in the continental United States (Merikangas et al., 2009). Interviews were administered in a dual‐frame sample that included household (n = 904) and school (n = 9244) sub‐samples. The NCS‐A response rate in the household sample was 85.9% (conditional on adult participation) and in the school sample was 74.7% (conditional on school participation). Each respondent was given a $50 incentive for participation. NCS‐A data were weighted for within‐household probability of selection (only in the household sub‐sample) and for residual discrepancies on the basis of socio‐demographic and geographic variables between the samples and the population distributions of US residents in the 13–17 age range based on the 2000 Census. More information on NCS‐A design and weighting are reported elsewhere (Kessler et al., 2009a; Kessler et al., 2009b).

The NCS‐A clinical reappraisal study was completed with a quota sample of 347 adolescent respondents and their parents that is described in more detail elsewhere (Kessler et al., 2009c). As K‐SADS clinical reappraisal interviews were administered by telephone, this sample was restricted to adolescents living in households with telephones. In the NCS‐A, each respondent (adolescent and parent) was given a $50 incentive for participation in the clinical reappraisal survey (over and above the $50 incentive for participation in the main survey). Adolescents who met DSM‐IV/CIDI criteria for one or more relatively uncommon disorders (e.g. agoraphobia, bipolar disorder, panic disorder, substance dependence with abuse) were oversampled relative to respondents who met criteria only for more common disorders. The lowest sampling fraction was for a third stratum made up of respondents who did not meet criteria for any lifetime DSM‐IV/CIDI disorder.

Measures

As noted earlier, the CIDI is a fully‐structured diagnostic interview designed for administration by trained lay interviewers to provide clinical diagnoses reflecting criteria specified in the DSM‐IV. The CIDI was originally developed for adults, but was modified for use with adolescents in ways described in detail elsewhere (Merikangas et al., 2009). Briefly, diagnostic sections of the adult CIDI were reviewed to determine whether assessments of particular disorders should be included or removed (for example, Pathological Gambling was eliminated based on the presumed low prevalence among youth and to minimize respondent burden). The language in remaining CIDI sections was modified to enhance comprehension for adolescents, using an iterative process that involved cognitive debriefing interviews. CIDI modules were modified in content to make them more relevant to the contexts and experiences of adolescents (e.g. referring to school life and peer relationships rather than work life and parenting). Other changes included updating drug terminology (including slang terms) in the substance use module and adding more detailed assessments of symptoms in the modules for disorders of childhood and adolescence (i.e. Attention Deficit Hyperactivity Disorder, Oppositional Defiant Disorder, Conduct Disorder and Separation Anxiety Disorders). Changes in content were piloted in the adolescent cognitive interviews and were revised as necessary. The finalized revision of each diagnostic module was reviewed for meaning, logic, and comparability to the adult version. Each diagnostic section was then systematically piloted to test the flow and timing among adolescents, with subsequent modifications to reduce the length of the diagnostic sections.

After a series of warm‐up questions, the CIDI administers screening questions made up of diagnostic stem questions for a wide range of disorders. Positive responses are then probed in subsequent CIDI sections. Although the CIDI evaluates 30‐day, 12‐month, and lifetime disorder criteria, we focus here exclusively on lifetime diagnoses.

The CIDI uses a stem‐branch structure in which a small number of diagnostic stem questions are administered for each syndrome and follow‐up questions are administered only for respondents who endorse at least one of these stem questions. In the case of panic, two stem questions are used. The first asks whether the respondent ever had “an attack of fear or panic when all of a sudden you felt very frightened, nervous, or uneasy.” Respondents who respond negatively are then administered a second stem question, which asks if the respondent ever had an attack when “all of a sudden you became very uncomfortable, you became short of breath, dizzy, sick to your stomach, or your heart beat very fast, or you thought that you might lose control, die, or go crazy.”

The CIDI stem question for social phobia asks respondents if they ever had a time in their life when they “felt very afraid or really, really shy with people, like meeting new people, going to parties, going on a date, or using a public bathroom.” Respondents who deny this question are administered a second stem question, which asks if they ever had a time in their life when they felt very afraid or uncomfortable when they had to “do something in front of a group of people, like giving a speech or speaking in class.”

The CIDI stem question for specific phobia asks respondents if they ever had a time in their life when they “felt a lot more afraid than most people of any of the following things.” A list is then read that includes a series of stimuli that match the subtypes described in the DSM‐IV (animal, natural environment, blood‐injection‐injury, and situational). Finally, the CIDI stem question for agoraphobia asks respondents if there was ever a time in their life when they “felt very afraid of either being in crowds, going to public places, traveling by your self, or traveling away from home.”

Respondents who endorsed stem questions were them administered a series of follow‐up questions corresponding to specific DSM‐IV criteria for these disorders. While the CIDI asks about all diagnostic criteria for each disorder, not all criteria were operationalized for final diagnostic assignment. In particular, we initially excluded Criterion C for social and specific phobia (recognizes that fear is unreasonable) because the DSM indicates that this criterion may not be present in children and this criterion in the CIDI had the weakest concordance with the K‐SADS Criterion C for both diagnoses. In addition, for all disorders we excluded the criterion specifying that the disorder could not be better accounted for by another mental disorder, as the CIDI did not measure all DSM‐IV disorders.

The CIDI was validated against the K‐SADS, a semi‐structured clinician‐administered diagnostic interview. Clinicians administering the K‐SADS in this study completed training with an experienced K‐SADS trainer and one of the developers of the K‐SADS and were closely supervised throughout the study (Kessler et al., 2009c). Clinicians were blinded to CIDI results, with the exception of responses to the screening questions, as prior studies have indicated that respondents in community surveys tend to report less as they are interviewed more often because of response fatigue, leading to the biased perception that earlier interviews over‐estimate prevalence compared to later ones. To address this problem, clinical interviewers were informed of responses to diagnostic stem questions, but not whether respondents met full criteria for disorders. Because the vast majority of respondents who endorse CIDI stem questions do not go on to meet full DSM‐IV/CIDI criteria for the associated disorder, this partial un‐blinding of interviewers did not inform clinical interviewers whether the CIDI diagnosis was positive.

Clinical interviews were carried out initially with the adolescent and then with a parent. Reconciliation follow‐up interviews were then carried out with the adolescent when there were discrepancies between adolescent and parent reports. K‐SADS interviewers used clinical judgment to combine information from both respondents. A subsequent review of these diagnostic decisions indicated that clinicians almost always chose to use the “or” rule to create a composite of parent and adolescent data. Final diagnoses, which are considered here, use these combined diagnoses.

Analysis methods

Concordance of CIDI and K‐SADS diagnoses was evaluated in the clinical reappraisal sample after weighting the data to adjust for the over‐sampling of CIDI cases and post‐stratifying the weighted data to adjust for small residual discrepancies with the full NCS‐A sample on a wide range of matching variables. Aggregate concordance of prevalence estimates was compared using McNemar χ ² tests that took into account unequal sampling weights. Individual‐level concordance was evaluated treating K‐SADS diagnoses as the gold standard by calculating CIDI sensitivity (SN), specificity (SP), positive predictive value (PPV), negative predictive value (NPV), Cohen's k (Cohen, 1960), and the area under the receiver operating characteristic curve (AUC) of a CIDI diagnosis predicting the K‐SADS diagnosis. All analyses of concordance were conducted using SAS 9.0 and SUDAAN 9.0.1 software programs (Research Triangle Institute, 2005; SAS Institute, 2002).

The factor structure of social phobia fears was investigated with data from the 6920 adolescents who screened into the CIDI social phobia diagnostic section using tetrachoric factor analysis with Promax rotation. Item response theory (IRT) models (Hambleton et al., 1991) were used to evaluate the implicit assumption in the DSM‐IV that each social fear has the same association as the others with the underlying syndrome. As many respondents did not endorse any social phobia fear items, we estimated a two‐class mixture model (Finkelman et al., 2011) that conceptualizes respondents as falling into two mutually exclusive categories, the first consisting of respondents who would report no social fears even with extended probes and the second of respondents in the fear symptom spectrum whose responses are described adequately by an IRT model with normally distributed latent liability (Finkelman et al., 2011).We estimated the percentage of people in each category, and IRT slopes, thresholds, and factor analysis parameters in the second class.

Results

Validity

Substantial agreement was found between diagnoses of social phobia based on the CIDI and K‐SADS (k = 0.63; AUC = 0.88; Landis and Koch 1977) (Table 1). However, the CIDI over‐estimated the prevalence of social phobia compared to the K‐SADS (CIDI = 13.6%, K‐SADS = 9.2%; χ² ₁ = 19.4, p < 0.001). To determine whether this overestimation was related to particular diagnostic criteria, we compared criterion‐level prevalence estimates of the CIDI and the K‐SADS. At the criterion level, all CIDI estimates were significantly associated with the clinician K‐SADS (AUC ranged from 0.70 to 0.79). For each social phobia criterion, though, adolescent self‐report on the CIDI over‐estimated the clinician's rating (all McNemar χ ² tests significant at p < 0.05). To reduce CIDI diagnostic prevalence, we looked for ways to tighten diagnostic criteria. First, we examined whether we could improve the prevalence rate by re‐introducing Criterion C. However, we found that this addition was too restrictive, leading to a substantial reduction in sensitivity (36.4%) and AUC (0.67). Instead, we found that CIDI prevalence could be made more consistent with the K‐SADS by requiring seven or more social fears. Revising the CIDI diagnostic algorithm in this way brought the estimated prevalence of social phobia in the CIDI closer to the gold standard (9.8% versus 9.2%, χ ² ₁ = 0.3, p = 0.62). Although this modification reduced overall concordance (from k = 0.63 to 0.59 and from AUC = 0.88 to 0.81), we felt that the improvement in concordance of the prevalence estimate based on the CIDI with prevalence based on the K‐SADS justified this change. The proportion of CIDI positives confirmed by the K‐SADS (PPV) increased from 55.9% to 61.3% with this change in the CIDI diagnostic threshold, although the proportion of K‐SADS cases detected by the CIDI (SN) decreased from 82.9% to 65.5%.

Table 1.

Concordance (sensitivity and specificity) of CIDI/DSM‐IV social phobia diagnostic criteria with blinded K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 347)

						Concordance
	K‐SADS prevalence		Youth CIDI prevalence		McNemar test	SNb		SPb		PPVb		NPVb		κb	AUCb
	Percentage	(SE)	Percentage	(SE)	χ 2	Est.	(SE)	Est.	(SE)	Est.	(SE)	Est.	(SE)
Social phobia
Criterion A: Marked fear of social situation	15.0	(2.2)	40.6	(3.2)	54.2*	83.7	(8.0)	67.0	(3.5)	30.9	(4.1)	95.9	(2.3)	.30	.75
Criterion B: Exposure provokes anxiety	13.7	(2.0)	36.1	(3.4)	38.6*	74.6	(7.7)	70.1	(3.7)	28.4	(4.3)	94.5	(2.1)	.27	.72
Criterion C: Recognizes fear is unreasonable	11.5	(1.8)	18.9	(2.3)	11.6*	53.7	(9.0)	85.6	(2.2)	32.5	(5.4)	93.4	(1.8)	.31	.70
Criterion D: Feared situations avoided	12.9	(1.8)	43.9	(3.5)	77.4*	88.7	(5.7)	62.7	(4.0)	26.1	(3.9)	97.4	(1.4)	.25	.76
Criterion E: Impairment	13.9	(2.1)	18.2	(1.9)	4.2*	68.6	(9.1)	90.0	(1.7)	52.6	(6.5)	94.7	(2.0)	.52	.79
Criterion F: Persistence > six months	14.5	(2.0)	39.8	(3.2)	58.1*	84.6	(6.6)	67.8	(3.6)	30.8	(4.3)	96.3	(1.8)	.30	.76
Overall diagnosis: Criteria A, B, C, D, E, F	9.2	(0.5)	6.0	(0.4)	3.0	36.4	(2.5)	97.1	(0.3)	55.8	(3.2)	93.8	(0.4)	.40	.67
Overall diagnosis: Criteria A, B, D, E, F	9.2	(1.7)	13.6	(1.6)	19.4*	82.9	(7.8)	93.4	(1.3)	55.9	(7.3)	98.2	(1.0)	.63	.88
Modified diagnosis: Criteria A, B, D, E, F & seven or more fears	9.2	(1.7)	9.8	(1.4)	0.3	65.5	(9.2)	95.8	(1.2)	61.3	(9.3)	96.5	(1.3)	.59	.81

^*The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test

.

^bSN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; κ, Cohen's κ; AUC, area under the receiver operating characteristic curve, SE, standard error; Est., estimate.

Substantial agreement was found between CIDI and K‐SADS diagnoses of specific phobia (k = 0.73; AUC = 0.94) (Table 2) as well as for individual criteria for that diagnosis (AUC ranged from 0.67 to 0.73). As with social phobia, the prevalence estimate based on the CIDI generally over‐estimated prevalence based on the K‐SADS (19.2% versus 12.7%; χ ² ₁ = 20.7, p < 0.001). The high CIDI estimate led to an extremely high proportion of K‐SADS cases being detected (SN) in the CIDI (96.9%), but to a lower proportion of CIDI cases being confirmed (PPV) by the K‐SADS (63.9%). Unlike the case with social phobia, though, we were unable to develop a satisfactory modification of the CIDI diagnostic algorithm to bring the prevalence estimate based on the CIDI closer to the K‐SADS prevalence estimate. In particular, although restoring Criterion C to the diagnostic algorithm improved our prevalence estimate (15.0% versus 12.7%; χ ² ₁ = 2.9, p > 0.05), sensitivity and the area under the receive operating characteristic curve were substantially reduced (SN = 74.3%, AUC = 0.84). Because of this substantial diminution in individual‐level concordance, we returned to the algorithm without Criterion C, recognizing that it overestimates prevalence.

Table 2.

Concordance (sensitivity and specificity) of CIDI/DSM‐IV specific phobia diagnostic criteria with blinded K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 347)

						Concordance
	K‐SADS prevalence		Youth CIDI prevalence		McNemar test	SNb		SPb		PPVb		NPVb		κb	AUCb
	Percentage	(SE)	Percentage	(SE)	χ 2	Est.	(SE)	Est.	(SE)	Est.	(SE)	Est.	(SE)
Specific phobia
Criterion A: Marked and persistent fear	34.3	(3.8)	71.5	(2.3)	132.3*	95.7	(2.2)	41.1	(3.6)	45.9	(5.1)	94.8	(2.7)	.29	.68
Criterion B: Exposure provokes anxiety	31.5	(3.3)	46.4	(3.8)	5.1*	72.1	(5.4)	65.4	(4.3)	48.9	(5.6)	83.6	(3.2)	.33	.69
Criterion C: Recognizes fear is unreasonable	29.0	(3.1)	37.2	(3.3)	8.2*	61.4	(5.7)	72.7	(3.6)	47.9	(5.0)	82.2	(3.3)	.31	.67
Criterion D: Feared situations avoided	32.5	(3.5)	58.3	(3.0)	104.0*	88.7	(3.3)	56.3	(3.5)	49.5	(4.4)	91.2	(2.8)	.37	.73
Criterion E: Impairment	24.2	(3.0)	23.7	(3.0)	0.04	58.3	(5.9)	87.3	(2.5)	59.5	(6.1)	86.8	(2.4)	.46	.73
Criterion F: Persistence > six months	36.0	(3.8)	52.0	(3.1)	24.5*	79.6	(4.3)	63.6	(4.0)	55.2	(4.9)	84.7	(3.9)	.39	.72
Overall diagnosis: Criteria A, B, C, D, E, F	12.7	(0.5)	15.0	(0.6)	2.9	74.3	(2.0)	93.7	(0.4)	62.9	(1.9)	96.2	(0.3)	.63	.84
Overall diagnosis: Criteria A, B, D, E, F	12.7	(2.4)	19.2	(3.1)	20.7*	96.9	(2.8)	92.1	(2.2)	63.9	(7.3)	99.5	(0.5)	.73	.94

^*The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test

.

^bSN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; κ, Cohen's κ; AUC, area under the receiver operating characteristic curve, SE, standard error; Est., estimate.

Concordance between CIDI and K‐SADS diagnoses of agoraphobia without panic disorder was strong (k = 0.60; AUC = 0.90). CIDI prevalence was slightly higher than K‐SADS prevalence [2.6% versus 1.5%, p < 0.05, 95% confidence interval (CI) for difference = 0.006–0.015] (Table 3). The low estimated prevalence prevented us from studying concordance at the item or criterion level and prevented us from developing a modification to bring the prevalence estimates of the CIDI and the K‐ SADS even closer. As CIDI prevalence was higher than K‐SADS prevalence, a high proportion of K‐SADS cases were detected (SN) by the CIDI (81.9%), while a lower proportion of CIDI cases were confirmed (PPV) by the K‐SADS (48.9%).

Table 3.

Concordance (sensitivity and specificity) of CIDI/DSM‐IV panic disorder diagnostic criteria and agoraphobia with blinded K‐SADS ratings in the NCS‐A clinical reappraisal sample (n = 347)

						Concordance
	K‐SADS prevalence		Youth CIDI prevalence		McNemar test	SNb		SPb		PPVb		NPVb		κb	AUCb
	Percentage	(SE)	Percentage	(SE)	χ 2	Est.	(SE)	Est.	(SE)	Est.	(SE)	Est.	(SE)
Agoraphobia
Overall diagnosis	1.5	(0.7)	2.6	(0.6)	4.8*	81.9	(15.1)	98.7	(0.5)	48.9	(19.1)	99.7	(0.3)	.60	.90
Panic disorder
Criterion A1c	3.6	(0.8)	3.8	(0.8)	0.02	49.6	(11.0)	98.0	(0.7)	47.8	(13.8)	98.1	(0.5)	.47	.74
Criteria A1 & A2d	2.5	(0.7)	2.4	(0.5)	0.80	61.9	(13.0)	99.2	(0.4)	65.8	(17.0)	99.0	(0.4)	.63	.81
Overall diagnosis: A1, A2,& Ce	2.1	(0.7)	2.4	(0.5)	0.84	74.4	(13.8)	99.2	(0.4)	65.8	(17.0)	99.4	(0.4)	.69	.87

^*The prevalence estimate based on the CIDI differs significantly from the estimate based on the K‐SADS at 0.05 level, two‐sided test.

^bSN, sensitivity; SP, specificity; PPV, positive predictive value; NPV, negative predictive value; ĸ, Cohen's ĸ; AUC, area under the receiver operating characteristic curve, SE, standard error; Est., estimate.

^cCriterion A1: Recurrent unexpected panic attack.

^dCriterion A2: At least one attack has been followed by one month (or more) of (i) persistent concern about having additional attacks, (ii) worry about implications or consequences of attack, or (iii) a significant change in behavior related to attacks.

^eCriterion C: Attacks not due to substance or medical condition.

Finally, substantial agreement was found between CIDI and K‐SADS diagnoses of panic disorder (k = 0.69; AUC = 0.87) (Table 4). CIDI and K‐SADS prevalence estimates were similar to each other (2.4% versus 2.1%; χ ² ₁ = 0.8, p = 0.41). Three‐fourths of K‐SADS cases were detected (SN) by the CIDI (74.4%) and 65.8% of CIDI cases were confirmed (PPV) by the K‐SADS.

Table 4.

Social phobia fear symptom prevalence, unrotated factor loadings, standard 1PL IRT threshold parameters, mixture model 1PL IRT threshold parameters, and rotated (promax) tetrachoric factor analysis (standardized regression coefficients) among adolescents classified by the mixture model IRT analysis as being in the spectrum

Criteria	Prevalence (n = 10,123)a		Standard Factor Analysis (n = 6920)a	Standard 1PL IRT (n = 6920)a	Mixture Model 1PL IRT (n = 6920)a	Mixture Model Factor Analysisd (n = 4382)a
	Percentage	(SE)	Factor 1	Thresholdb	Thresholdc	Factor 1	Factor 2	Factor 3
Meeting new people	23.6	(0.8)	0.86	0.58	−0.23	0.90	−0.11	−0.07
Talking to authority	20.3	(0.8)	0.82	0.73	0.13	0.48	0.18	0.21
Speaking in class	24.9	(0.8)	0.82	0.52	−0.37	0.09	0.28	0.57
Going to parties	8.7	(0.3)	0.70	1.35	1.55	0.80	−0.00	−0.24
Performing for audience	35.8	(0.9)	0.87	0.02	−1.81	−0.11	−0.10	0.90
Taking important exam	19.8	(0.9)	0.73	0.75	0.18	0.06	0.41	0.22
Working while someone watches	14.1	(0.6)	0.72	1.03	0.83	−0.10	0.85	0.03
Going into a room	16.1	(0.7)	0.81	0.93	0.59	0.53	0.28	0.05
Talking to strangers	22.2	(0.8)	0.86	0.64	−0.08	0.73	−0.02	0.16
Disagreeing with people	16.0	(0.7)	0.75	0.93	0.60	0.46	0.11	0.19
Doing things someone watches	14.5	(0.6)	0.76	1.01	0.78	0.03	0.81	−0.02
Using public bathroom	10.3	(0.4)	0.67	1.25	1.32	0.14	0.62	−0.15
Going out/dating	13.3	(0.7)	0.74	1.07	0.92	0.58	0.07	0.05
Situation that could be embarrassed	24.6	(0.8)	0.81	0.53	−0.34	0.42	−0.10	0.46

^aSample sizes reflect the full CIDI sample (n = 10,123), the sub‐sample of adolescents who screened into the social phobia diagnostic interview section (n = 6920), and the smaller sub‐sample of respondents who screened into the social phobia section who were estimated to be described adequately by the normally distributed latent liability assumed in the IRT model (n = 4382).

^bAll item slopes = 1.5.

^cAll item slopes = 0.6.

^dCorrelations among factors: F1 & F2 = 0.44, F1 & F3 = 0.22, F2 & F3 = 0.21 (F1, Factor 1; F2, Factor 2; F3, Factor 3). Highest factor loadings are indicated in italic typeface.

The structure of social fears

As noted earlier, we were particularly interested in the structure of social phobia fear symptoms. The tetrachoric factor analysis of social phobia fears in all adolescents who screened into the section (n = 6920) suggested a unidimensional solution (with unrotated eigenvalues of the first two principal factors of 8.5 and 1.0). Factor loadings were high and fairly similar across fears (0.67–0.87) (Table 1). In IRT models, thresholds varied considerably, ranging from 0.02 for fear of performing to 1.4 for fear of going to parties. Likelihood ratio tests did not support selection of the 2PL IRT model over the more parsimonious 1PL model, suggesting that the different fears had equal discriminating power at their thresholds so an equally‐weighted symptom count is an appropriate scoring rule.

As a high proportion of adolescents screened into the social phobia section, but did not endorse any social phobia fear symptoms, a two‐class IRT mixture model was estimated and found to be a better fit to the data than the traditional 1PL IRT model (χ ² ₁ = 11,954, p < 0.001). This model estimated that 38% of adolescents were outside (below) the social phobia spectrum. Excluding the respondents outside the spectrum reduced item thresholds and further increased the range of these thresholds (−1.8 to 1.6). A tetrachoric factor analysis restricted to respondents who were in the spectrum was much more differentiated and suggested a three‐factor solution (with unrotated eigenvalues of 4.7, 1.3, 1.3, and 0.9). One factor corresponded to social or interactional fears (meeting new people, going to parties, talking to strangers), while the other two included indicators of performance fears, one having heavy loadings for fears of public speaking and performance and the other for being evaluated or watched while doing activities.

Discussion

We found that DSM‐IV diagnoses of panic and phobia from the adolescent version of the CIDI for the most part had good concordance with independent clinical diagnoses based on the K‐SADS. However, the CIDI generally over‐estimated prevalence compared to the K‐SADS. We were able to address this problem in the case of social phobia by modifying the CIDI diagnostic threshold, but no comparable change was possible in the case of specific phobia or agoraphobia. Data on specific phobia and agoraphobia from the adolescent CIDI should consequently be interpreted cautiously, recognizing that prevalence is over‐estimated.

Our finding that social phobia fear symptoms form a unidimensional scale is consistent with several other studies suggesting that social fears are best conceptualized as a continuum without subtypes (Ruscio et al., 2008; Vriends et al., 2007). However, a new factor analysis excluding adolescents outside the social phobia spectrum suggested a more differentiated three‐factor structure. The distinction between social/interactional and performance fears has been described by others (Kessler et al., 1998; Perugi et al., 2001) and suggests that these may be discrete subtypes of social phobia in adolescents (particularly the distinction between social fears and fears of public performance). While we considered weighting the fear symptom factors that emerged in our tetrachoric factor analysis, the CIDI already emphasizes social/interactional fears and we were concerned that further down‐weighting of other fears would diminish aspects of social phobia that we believe to be important. In addition, although the DSM‐IV does not recommend summing fear symptoms, there is evidence that functional impairment increases with the number of fears endorsed, regardless of their content (Ruscio et al., 2008; Stein et al., 2000). In our data, requiring the increased number of fear symptoms corresponded to an improvement in diagnostic concordance with the K‐SADS. The inclusion of clinical judgment in the K‐SADS may mean that clinician ratings incorporate impressions of clinical severity to a greater extent than in the fully‐structured CIDI interview. Future research may profitably investigate the possibility of using information about fear symptoms to refine structured assessments of social phobia. Although the adolescent version of the CIDI had a longer list of fear symptoms than the CIDI used in the adult sample of the National Comorbidity Survey, which had only six symptoms (Kessler et al., 1998), it is also possible that measurement could be improved further with a longer and more heterogeneous list of fears.

Limitations of the design of the clinical reappraisal study include administration of the K‐SADS over the telephone rather than in person. While validity studies that administer surveys over the phone have shown concordance that is equivalent to face‐to‐face surveys (Rohde et al., 1997), by administered the K‐SADS by telephone and the CIDI in person we introduced an additional method effect. Another potential limitation is that the CIDI diagnoses for these disorders were based only on adolescent report, while the clinician K‐SADS diagnoses combined information from parent and adolescent interviews. The generally poor agreement of parents and adolescents on measures of anxiety (Comer and Kendall, 2004; Grills and Ollendick, 2003) suggests that the inclusion of parent data on the K‐SADS may have reduced concordance between the CIDI and K‐SADS. However, we tested this possible bias by examining the concordance of CIDI responses with K‐SADS diagnoses based on the adolescent report alone. For all disorders, concordance was weaker when the K‐SADS did not incorporate parent response (for panic, AUC = 0.74, for specific phobia, AUC = 0.64, for social phobia, AUC = 0.69), indicating that parent report contributed to diagnostic validity. As multi‐informant assessments are generally preferred over single‐informant assessments, it is important that we used a multi‐informant interview as the gold standard.

Declaration of interest statement

Competing interests: Dr Kessler has been a consultant for GlaxoSmithKline Inc., Kaiser Permanente, Pfizer Inc., Sanofi‐Aventis, Shire Pharmaceuticals, and Wyeth‐Ayerst; has served on advisory boards for Eli Lilly & Company and Wyeth‐Ayerst; and has had research support for his epidemiological studies from Bristol‐Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Johnson & Johnson Pharmaceuticals, Ortho‐McNeil Pharmaceuticals Inc., Pfizer Inc., and Sanofi‐Aventis. The remaining authors report no competing interests.