A comparison between screened NIMH and clinically interviewed control samples on neuroticism and extraversion

Abstract

The National Institute of Mental Health (NIMH) has supported the collection of DNA samples on over 4000 subjects for use primarily as controls in psychiatric genetic studies. These subjects, though screened online, were not directly interviewed or assessed on family history. We compared this sample to one that was directly interviewed using structured diagnostic assessments on comparable measures of neuroticism and extraversion. The screened sample completed an online self-report based on the Composite International Diagnostic Instrument Short-Form (CIDI-SF). The interviewed sample was assessed by clinically trained personnel using the Schedule for Affective Disorders and Schizophrenia (SADS-LA-IV) and Family History Screen; final diagnoses were made blind to trait scores by a clinician using the best-estimate procedure. Neuroticism and extraversion were assessed on the NEO five-factor inventory (NEO-FFI) and the revised Eysenck Personality Questionnaire short form (EPQ-R). We found that subjects in the NIMH-screened sample who did not report any psychiatric symptoms on the self-report were indistinguishable from interviewed diagnosis free and family history negative controls on neuroticism and extraversion. Subjects in the screened sample who screened positive for anxiety disorders, however, deviated significantly on these measures both from the screened subjects with no self-reported symptoms, as well as from subjects in the interviewed sample diagnosed with comparable disorders. These findings suggest that control groups generated from the NIMH sample should ideally be restricted to subjects free of any self-reported symptoms, regardless of the disorder being addressed, in order to maximize their reflection of diagnosis-free populations.

Introduction

Genetic studies of complex psychiatric disorders require large sample size in order to achieve adequate statistical power.^1,2 In the late 1980s, the National Institute of Mental Health (NIMH) launched the Human Genetics Initiative (http://nimhgenetics.org/) to support the systematic collection of large samples of DNA and clinical data, and to make them available in repositories for sharing with other qualified investigators. Although initially targeting families with schizophrenia, bipolar disorder and Alzheimer’s disease, the initiative has been progressively expanded to include other psychiatric disorders.^3–5 To enable association (case-control) studies, DNA samples from more than 4000 unrelated subjects (here-tofore, ‘screened sample’) have also been collected and genotyped, and are now being used with increasing frequency by a number of investigators. As the availability of this sample is recent, only one study to our knowledge has been published.⁶

These subjects were screened online using a self-report based on the Composite International Diagnostic Interview Short-Form (CIDI-SF).⁷ However, they were not directly interviewed or diagnosed by a clinician, only select psychiatric domains were assessed and no family history was ascertained. Subjects who endorsed CIDI-SF symptoms were not excluded from the repository sample, but further exclusion criteria could be imposed by individual investigators based on desired population characteristics.

We recently used these screened controls in a case-control genetic study of panic disorder (PD) and social anxiety disorder (SAD) (Talati et al., in press). As part of that study, we also recruited a parallel control sample with no lifetime history of any psychiatric disorder, as well as no history of any anxiety disorder in any first- or second-degree relative (heretofore, ‘interviewed sample’). These non-ill controls, as well as our PD and SAD subjects, were directly interviewed using structured diagnostic assessments,⁸ and final diagnoses were made by experienced clinicians blind to genetic data using the best-estimate procedure.⁹

In this report, we compare the screened sample with our interviewed sample on shared measures in order to address clinical comparability. Specifically, we examine two traits from the five-factor model (FFM) of personality trait organization, neuroticism (the degree of proneness to experience negative emotional states) and extraversion (proneness to social interaction).¹⁰ These traits have been associated with emotional well-being, health-related quality of life and psychiatric symptoms, notably those along the mood-anxiety spectrum.^11,12 There is a body of evidence documenting that subjects with anxiety disorders, for example, tend to score significantly higher than the mean population on the neuroticism scale, and conversely, lower (particularly among those with phobias) on the extraversion scale.^13–15 These quantitative personality traits may therefore serve as useful stand-ins of symptom severity in the absence of formal diagnoses, as well as provide refinements within existing categorical definitions.

The primary purpose of this report is to compare the subjects in the screened NIMH sample to our interviewed controls on these two traits. Because the NIMH sample also included some subjects who self-screened for anxiety disorders, we further compare these with subjects in the interviewed sample who were formally interviewed and diagnosed with comparable disorders.

Materials and methods

The interviewed sample was obtained from a genetic case-control study of PD and SAD (Talati et al., in review). Probands were required to be at least 18 years of age at the time of assessment, and to have a probable or definite DSM-IV diagnosis of PD (with or without agoraphobia), or generalized SAD, with a family history of anxiety in at least one first-degree relative. Because earlier age of onset has been shown to be associated with increased familial aggregation and longitudinal impairment,^16,17 we also required first onset by age 30. Probands with a history of bipolar disorder, schizophrenia or anti-social personality were excluded a priori, as were those for whom a medical or neurological disorder, or use of a substance, was judged to be the cause of the anxiety symptoms. Controls were required to be at least 30 years of age (to minimize the possibility of capturing those who were still at risk for developing these disorders), and to have no evidence of any psychiatric disorder over their lifetime (excepting bereavement, adjustment disorders and substance abuse in adolescence, college or the military), as well as no history of any anxiety symptoms in any first- or second-degree relative. All procedures were approved by the New York State Psychiatric Institute’s Institutional Review Board, and all subjects gave written consent prior to participation.

Assessments and training procedures have been detailed previously (Talati et al., in press). Briefly, diagnoses were ascertained by masters and doctoral level mental health professionals using the Schedule for Affective Disorders and Schizophrenia-Lifetime Version modified for the study of anxiety disorders and updated for DSM IV (SADS-LA IV).⁸ Psychiatric history on first-degree relatives was obtained from the proband using the Family History Screen (FHS).¹⁸ Final psychiatric diagnoses were made by a clinically trained person based on all available diagnostic information using the best-estimate procedure.⁹ The person determining the final diagnosis was uninvolved in the interviewing, and was blind to genetic data and trait scores. Because the NIMH-screened sample (see below) included only non-Hispanic Caucasian and African-American subjects, we excluded subjects in the Hispanic and ‘other’ categories (6%) in these analyses, yielding a final sample of 190 probands with PD, 172 with SAD, 153 with both PD and SAD, and 90 normal controls.

Screened sample

Subject DNA and screen information were provided by the Rutgers University Cell and DNA repository (www.rucdr.org) as part of the NIMH Human Genetics Initiative. Subjects had been initially recruited by a marketing company under sub-contract to the NIMH using modern research survey methods to obtain a representative sample of the population of European and African-American ancestry. Subjects were assessed using an online self-report based on the CIDI-SF (lifetime version, modified for DSM-IV).⁸ Derived from the longer Composite International Diagnostic Interview (CIDI), the CIDI-SF is a structured set of scales assessing eight syndromes: major depressive episodes, panic attacks, agoraphobia, social phobia, specific phobia, generalized anxiety disorder (GAD), obsessive-compulsive disorder (OCD) and alcohol and drug dependence. Each diagnostic module was configured in a stem-branch structure, and subjects only completed the section if they endorsed the initial screening questions. In addition, subjects were also asked if they had ever been diagnosed with, or received treatment for, schizophrenia, schizoaffective disorder or bipolar disorder. The CIDI-SF had 90% sensitivity/95% specificity for PD and 86/99% for social phobia to the CIDI in a US National Comorbidity Survey.^7,19,20

Clinical and genetic information was provided for the 2959 recruited subjects who were 18–75 years of age and free of schizophrenia, schizoaffective disorder and bipolar disorder. Of these, we designated 534 subjects who were above 30 years of age and screened negative for all assessed psychiatric disorder categories on the CIDI-SF as the control group. Dichotomous PD and SAD categories were generated from the remaining subjects using CIDI-SF criteria for probability of caseness, as described by Walters et al.²¹ A total of 90 subjects met criteria for PD, 136 for SAD and 10 for both PD and SAD. The sample included 87% Caucasian, and 13% African-American subjects.

To test for variation in applicability across the SADS-LA and CIDI-SF, we also regenerated diagnoses in the interviewed sample by applying the CIDI-SF criteria from the screened sample to corresponding SADS items. Sensitivity of these regenerated diagnoses was 100% for both PD and SAD; that is, all subjects who had been originally assigned a diagnosis of PD or SAD on the SADS-LA interview also met the CIDI-SF criteria for the same disorder. However, because the interviewed sample required employed additional criteria that the screened sample did not (namely, first onset by age 30, four or more attacks in 4 weeks (for PD), evidence of generalizability,²² (for SAD)), specificity of the regenerated diagnoses was lower: 20 (11%) subjects in the interviewed sample who had not originally met criteria for SAD, now did so; conversely, 24 (14%) subjects who did not meet PD criteria in the interviewed sample now qualified for the PD group. After accounting for the above variation in criteria, however, there was complete diagnostic concordance between the SADS-LA and CIDI-SF.

Neuroticism and extraversion traits in the interviewed sample were assessed using the neuroticism-extraversion-openness (NEO) personality inventory.²³ This self-report operationalizes a hierarchical structure based on the FFM of personality (neuroticism, extraversion, openness, agreeableness and conscientiousness), and has been repeatedly demonstrated to have high test–retest reliability, construct and predictive validity, heritability and evidence for cross-cultural universality.^24,25 The scale can be administered to anyone aged 17 years or older, and so was appropriate for all subjects. We used the abbreviated 60-item NEO five-factor inventory (NEO-FFI) version, ²³ in which 12 questions for each factor are rated on a 5-point scale with responses ranging from strongly agree to strongly disagree. The NEO-FFI is highly correlated with the parent 240-item NEO (0.77–0.92); correlations are highest for neuroticism (> 0.90). The inventory was completed by 186 (98%) PD, 166 (96%) SAD, 151 (96%) PD + SAD and all control probands.

The screened sample was assessed online using the 12-item short scale of the revised version of the Eysenck Personality Questionnaire (EPQ-R).²⁶ The EPQ-R neuroticism and extraversion scales each comprise 12 yes/no questions, and have been shown to be comparable to corresponding NEO-FFI scales.²⁷ Reliability (α-coefficients) for the neuroticism scale is 0.88 for males and 0.84 for females; for extraversion, 0.84 for males and 0.80 for females. Raw scale scores for the NEO-FFI and EPQ-R were each converted in to T-scores (scale 0–100), based on published population mean scores and standard deviations by age and gender, for each instrument. For both scales, higher scores indicate greater neuroticism and extraversion.

A flow chart illustrating the generation of the final interviewed and screened samples are outlined in Figure 1. Because only 10 subjects in the generated screened sample met criteria for both PD and SAD, we present the data using only the PD- and SAD-only diagnostic groups, alongside the controls.

Figure 1.

Flow chart of interviewed and screened samples.

Statistical analyses

All statistical analyses were performed in Statistical Analysis Software (SAS) Version 9.0 (Cary, NC, USA). (Parametric tests were used for all analyses as we found no significant evidence for skewness or kurtosis; however, the main findings shown in Figures 2 and 3 were also replicated using a non-parametric statistical model). Differences in demographic characteristics between were examined using χ² tests for categorical variables (or Fisher’s Exact test, if one or more cell had an expected cell count of less than 5), and an analysis of variance (ANOVA) for age. Differences in neuroticism and extraversion T-scores by sample and diagnostic group were tested using a general linear model (PROC GLM), with the neuroticism/extraversion scores as the continuous dependent variable, a six level variable representing the three diagnostic groups and two samples as the risk factor of interest, and age (continuous), sex and race as covariates. Contrast matrices were included in the model to enable comparisons (1) between the PD and SAD groups to corresponding control groups within each sample, and (2) within each group (PD, SAD, control) across samples. To correct for multiple comparisons (nine tests), an α-cutoff of 0.005 was selected a priori. A separate model was also run replacing the six-level risk variable with separate two-level sample and three-level group, variables as well as a sample × group term to test for interaction effects.

Figure 2.

Distribution of neuroticism by diagnostic group in interviewed and screened samples. Bars illustrate mean neuroticism scores for each group, adjusted for age, sex and race; error bars represent 99.5% confidence intervals. Higher scores indicate greater neuroticism. ^aMean differences reflect the absolute score difference between each comparison group adjusted for age, sex and race.

Figure 3.

Distribution of extraversion by diagnostic group in interviewed and screened samples. Bars illustrate mean extraversion scores for each group, adjusted for age, sex and race; error bars represent 99.5% confidence intervals. Higher scores indicate greater extraversion. ^aMean differences reflect the absolute score difference between each comparison group adjusted for age, sex and race.

Results

Demographics

Subject demographics are shown in Table 1, for both sample (interviewed and screened) and each diagnostic group (PD, SAD, control). Within each sample, controls were older than anxiety probands (by design, as they had to be > 30 years of age to ensure they had passed the risk period for first onset) and also included a significantly greater percentage of male, as well as married subjects. There were no group-sample interactions, indicating that the distribution of demographic variables across diagnostic categories did not vary significantly between the two samples. Finally, there were no significant differences in demographic variables between subjects who completed the assessment of neuroticism and extraversion, and those who did not.

Table 1.

Comparison of interviewed and screened samples on demographic characteristics

	Interviewed sample			P-valuea	Screened sample			P-valuea
	PD N (%)	SAD N (%)	Control N (%)		PD N (%)	SAD N (%)	Control N (%)
Age (mean, years)	39 (12)	40 (13)	51 (14)	F = 29, d.f. = 2, P < 0.0001	43 (13)	47 (14)	54 (13)	F = 39, d.f. = 2, P < 0.0001
N, % Female	154 (81)	126 (73)	51 (57)	χ2 = 18.5, d.f. = 2., P < 0.0001	68 (75)	81 (60)	266 (50)	χ2 = 22.0, d.f. = 2., P < 0.0001
Race (N, %)b
Caucasian	172 (92)	157 (93)	81 (90)	χ2 = 0.73, d.f. = 2., P = 0.69	78 (86)	119 (88)	454 (85)	χ2 = 0.6, d.f. = 2., P = 0.70
African-American	14 (8)	12 (7)	9 (10)		12 (13)	17 (13)	80 (15)
Education (N, %)
< High school	7 (4)	13 (7)	3 (3)	χ2 = 5.6, d.f. = 4., P = 0.23	9 (10)	16 (12)	30 (6)	χ2 = 17.3, d.f. = 4., P = 0.002
Some college	73 (38)	61 (35)	27 (30)		25 (28)	41 (30)	108 (20)
≥College degree	110 (58)	98 (57)	60 (67)		56 (62)	79 (58)	396 (74)
Marital status (N, %)
Single	96 (51)	90 (53)	22 (24)	χ2 = 23.6, d.f. = 3., P < 0.0001	22 (24)	35 (25)	56 (10)	χ2 = 33.6, d.f. = 4., P < 0.0001
Married	54 (29)	56 (33)	42 (47)		48 (53)	71 (52)	383 (72)
Separated, divorced or widowed	38 (20)	25 (15)	26 (29)		20 (22)	30 (22)	95 (18)
Employment (N, %)
Full or part time	122 (67)	97 (57)	64 (71)	χ2 = 5.7, d.f. = 2., P = 0.06	55 (61)	76 (55)	291 (55)	χ2 = 1.3, d.f. = 2., P = 0.5
Not employed	61 (33)	72 (43)	26 (29)		35 (39)	60 (44)	243 (45)

Abbreviations: ANOVA, analysis of variance; PD, panic disorder only; SAD, social anxiety disorder only.

^aP-values based on an analysis of variance (ANOVA) for age, and a χ² test for all other variables.

^bAll subjects were non-Hispanic (see Materials and methods section).

Distribution of neuroticism and extraversion traits

Distribution of trait scores is shown in Figures 2 (neuroticism) and 3 (extraversion). Within each sample, after adjusting for age, sex and race, subjects with either PD or SAD had significantly higher neuroticism scores than corresponding controls (Figure 2, see mean difference column), and significantly lower extraversion scores than the controls (Figure 3). Probands with SAD were further distinguishable (increased neuroticism scores, and decreased extraversion scores) from those with PD in the interviewed sample, but not in the screened sample.

We next compared the traits across samples to determine comparability. Probands with PD or SAD in the interviewed sample had significantly higher neuroticism and (for SAD) lower extraversion, than the comparable probands meeting criteria for the same disorders in the screened sample. However, importantly, the control groups of the two samples were not distinguishable from each other on either trait. There was also a significant diagnostic group-sample interaction for both neuroticism and extraversion traits, reflecting that the magnitude of the difference in trait scores (that is, the effect size) between anxiety and control probands was significantly greater in the interviewed sample. The effects were invariant to any differences in proband age, sex or race, and remained significant following regeneration of the PD and SAD groups in the interviewed sample using the CIDI-SF criteria such that all subjects had been assessed on the same instrument (see Materials and methods section).

We also repeated the above analyses, replacing the groups in the screened sample that had met criteria for PD and SAD (that is, columns 4 and 5 of Figures 2 and 3) with subjects who had endorsed the CIDI-SF gate questions for PD or SAD, but had not met full diagnostic criteria for inclusion. These analyses replicated the above findings, indicating that even subjects who met minimum screening criteria differed significantly on neuroticism and extroversion from the interviewed controls (data not shown but are available upon request).

Finally, we examined the effects of other psychiatric conditions on neuroticism and extraversion traits. Because all subjects in the interviewed sample were required to have a diagnosis of PD or SAD, we could only compare comorbidity with these disorders. After adjusting for age sex and race, lifetime depression (whether with PD or SAD) was associated with increased neuroticism in both the interviewed and the screened sample (Table 2). Having an additional anxiety disorder, whether GAD, specific phobia (SP) or OCD was associated with increased neuroticism only in the screened sample, but not in the interviewed sample. The findings shown in Figures 2 and 3 remained significant after adjusting for the presence of each of these disorders, either individually or in a combined model.

Table 2.

Comparison of neuroticism and extraversion by comorbid psychiatric conditions in the interviewed and screened samples

		Neuroticism				Extraversion
		Interviewed samplea		Screened sampleb		Interviewed sampleb		Screened sampleb
		PD	SAD	PD	SAD	PD	SAD	PD	SAB
MDD	Yes	68.1 (8)e	72.1 (11)d	56.5 (11)c	59.7 (9)e	39.9 (13)d	33.8 (12)	45.6 (12)	43.7 (9)
	No	60.8 (11)	65.4 (12)	50.3 (8)	49.3 (10)	46.7 (11)	35.7 (11)	50.4 (10)	48.0 (10)
GAD	Yes	67.0 (13)	70.7 (11)	58.1 (10)f	60.0 (9)c	43.9 (9)	32.9 (9)	46.1 (8)	43.7 (9)
	No	62.9 (10)	66.8 (9)	50.9 (9)	47.0 (9)	44.3 (10)	35.4 (11)	48.7 (9)	50.8 (10)
SP	Yes	64.8 (10)	67.5 (11)	60.1 (9)g	61.4 (8)c	44.6 (13)	35.6 (10)	43.7 (10)	42.8 (9)
	No	62.6 (11)	67.1 (11)	53.0 (11)	51 (10)	44.1 (12)	35.1 (11)	48.4 (10)	46.7 (9)
OCD	Yes	8.8 (10)	74.5 (9)c	65.1 (4)h	63.6 (7)c	43.3 (13)	34.4 (10)	42.5 (12)	41.3 (9)
	No	63.1 (9)	66.6 (11)	51.7 (11)	52.2 (11)	44.3 (11)	35.2 (10)	49.4 (12)	46.5 (10)

Abbreviations: CIDI-SF, Composite Diagnostic International Diagnostic Instrument Short-Form; GAD, generalized anxiety disorder; MDD, major depressive disorder; OCD, obsessive compulsive disorder; panic disorder; SAD, social anxiety disorder; SADS-LA, Schedule for Affective Disorders and Schizophrenia, Lifetime Version modified for Anxiety Disorders; SP = specific (simple) phobia.

^aInterviewed on the SADS-LA.

^bScreened on the CIDI-SF.

^cP < 0.0001;

^dP = 0.0015;

^eP = 0.02;

^fP = 0.0005;

^gP = 0.01;

^hP = 0.002.

P-values represent differences between probands with and without the condition listed in the left column, adjusting for age, sex and race.

Discussion

We compared an NIMH repository population that was screened using an online self-report to a population that was directly interviewed by clinically trained assessors, and who passed best-estimate procedures for a diagnosis. Our report demonstrates that (1) subjects without any self-reported symptoms in the screened sample were indistinguishable from the clinically interviewed control group on either neuroticism or extraversion; (2) subjects in the screened sample who met CIDI-SF criteria for anxiety disorders had lower neuroticism and higher extraversion scores than subjects who had been formally interviewed and assigned the same diagnosis, but conversely had higher neuroticism and lower extraversion scores than subjects with no symptoms (3) the difference in neuroticism and extraversion scores between subjects with and without anxiety (that is, the ‘effect size’) was greater in the clinically interviewed population. These patterns were invariant to the subject’s age, sex and race, comorbid depression or anxiety as well as to the diagnostic criteria employed.

These findings suggest that the subset of the repository sample with no discernable symptoms on the CIDI-SF may be representative (on assessed measures) of healthy controls who have never been mentally ill and have no family psychiatric history. This is an important ascertainment, as the NIMH subjects have never themselves been directly interviewed. Subjects who meet CIDI-SF criteria for anxiety disorders, however, if included in control groups (particularly when used in studies of anxiety or other disorders that co-segregate with it) could confound the studies, as we found them to deviate significantly on neuroticism and extraversion from non-ill controls. In fact, further inspection of the CIDI-SF revealed that for each covered diagnostic module depression, generalized anxiety, specific phobia, agoraphobia, obsessive-compulsive disorder and alcohol/drug dependence), subjects who either endorsed gate questions or met diagnostic criteria had significantly higher neuroticism and lower extraversion than the non-ill controls. Excluding individuals with any psychiatric symptoms from control groups, regardless of the disorder group addressed, may therefore be the most conservative approach.

Imposing such criteria would substantially reduce the available pool, which coupled with other restrictions of genetic studies (for example, matching) may necessitate trade-offs between having a larger sample versus greater homogeneity. In the case of replication studies, an increasingly vital component of psychiatric genetic research,²⁸ the sample must also be further matched to the original for comparability. Finally, investigators should note that a significant number of subjects in the NIMH repository sample were young at the time of the screen. Because the bulk of psychiatric disorders tend to have their first onset in adolescence or early adulthood, for example,29–31 persons under 30 might optimally be excluded from non-ill control groups to minimize the likelihood of including subjects who have not passed through the age of risk and who might later develop the illness.

Our findings showed that subjects meeting CIDI-SF criteria for anxiety introduce heterogeneity into control groups. At the same time, our data also suggest that such subjects should ideally not be used to generate anxiety disorder groups, as they may represent a less severe phenotype than those meeting our rigorous interview criteria, which could possibly lead to diminished effect sizes (note the smaller difference between anxiety and control trait scores in the screened, as compared to the interviewed sample, in Figures 2 and 3). The lower scores in the screened, vis-a-vis the interviewed, samples were not attributable to differences in diagnostic instruments, and likely reflect intrinsic variation among the two populations (such as differences in age of onset, frequency/severity of attacks and so on). Because these variables were not assessed in the NIMH sample, we cannot examine this argument further. The lack of family history and onset information in the screened sample however bears note, as earlier age of onset has been found to be predictive of greater familial aggregation,^16,17 and forms of disorders that aggregate within families may be genetically distinct from those that do not.³² The unavailability of such information may limit the ability to construct reliably homogenous and heritable disorder groups from the screened sample.

Finally, it is important that the findings of this report be interpreted cautiously. Because only two traits were assessed (there were no other comparable measures across the two samples), the findings of equivalence among the two control groups cannot be generalized, as it is possible that the populations differed substantially on other unmeasured characteristics. Furthermore, neuroticism and extraversion, despite evidence for heritability,^33–35 are not genetic markers, and our findings do not address genetic homogeneity. Third, because of the design of the interviewed sample, comparisons across disorders were restricted to panic and social anxiety. For other disorder groups with different comparative reliabilities for clinically assessed versus self-reported data, comparability may quite vary. Despite these limitations, no other study to our knowledge has directly compared the screened repository sample to a fully characterized interviewed population. The repository sample is being used and requested with increasing frequency, especially now that whole genome data are available, and this report highlights comparability across two important traits associated with psychiatric functioning.