Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Dec 1.
Published in final edited form as: Neuroinformatics. 2011 Dec;9(4):321–333. doi: 10.1007/s12021-010-9094-6

Neuropsychological Testing and Structural Magnetic Resonance Imaging as Diagnostic Biomarkers Early in the Course of Schizophrenia and Related Psychoses

Elissaios Karageorgiou 1,2,3,4, S Charles Schulz 5,6, Randy L Gollub 7, Nancy C Andreasen 8, Beng-Choon Ho 9, John Lauriello 10, Vince D Calhoun 11,12, H Jeremy Bockholt 13, Scott R Sponheim 14,15,16,17, Apostolos P Georgopoulos 18,19,20,21,22,✉
PMCID: PMC3116989  NIHMSID: NIHMS272182  PMID: 21246418

Abstract

Making an accurate diagnosis of schizophrenia and related psychoses early in the course of the disease is important for initiating treatment and counseling patients and families. In this study, we developed classification models for early disease diagnosis using structural MRI (sMRI) and neuropsychological (NP) testing. We used sMRI measurements and NP test results from 28 patients with recent-onset schizophrenia and 47 healthy subjects, drawn from the larger sample of the Mind Clinical Imaging Consortium. We developed diagnostic models based on Linear Discriminant Analysis (LDA) following two approaches; namely, (a) stepwise (STP) LDA on the original measurements, and (b) LDA on variables created through Principal Component Analysis (PCA) and selected using the Humphrey-Ilgen parallel analysis. Error estimation of the modeling algorithms was evaluated by leave-one-out external cross-validation. These analyses were performed on sMRI and NP variables separately and in combination. The following classification accuracy was obtained for different variables and modeling algorithms. sMRI only: (a) STP-LDA: 64.3% sensitivity and 76.6% specificity, (b) PCA-LDA: 67.9% sensitivity and 72.3% specificity. NP only: (a) STP-LDA: 71.4% sensitivity and 80.9% specificity, (b) PCA-LDA: 78.5% sensitivity and 91.5% specificity. Combined sMRI-NP: (a) STP-LDA: 64.3% sensitivity and 83.0% specificity, (b) PCA-LDA: 89.3% sensitivity and 93.6% specificity. (i) Maximal diagnostic accuracy was achieved by combining sMRI and NP variables. (ii) NP variables were more informative than sMRI, indicating that cognitive deficits can be detected earlier than volumetric structural abnormalities. (iii) PCA-LDA yielded more accurate classification than STP-LDA. As these sMRI and NP tests are widely available, they can increase accuracy of early intervention strategies and possibly be used in evaluating treatment response.

Keywords: Schizophrenia, Schizophreniform, Schizoaffective, PCA, LDA, Biomarkers, Neuropsychology, MRI, Cross-validation, Diagnosis, MCIC

Introduction

Making an accurate diagnosis after a first psychotic episode is important for initiating treatment and counseling patients and families (Harrigan et al. 2003). Classification systems for psychiatric disorders are distinguished from the classification of other diseases by their major reliance on behavioral signs and symptoms. Nearly 40 years ago Robins and Guze (1970) outlined a means for developing laboratory tests for mental disorders in order to improve the reliability and validity of classification and move diagnostic assessment beyond clinical description. They suggested that, following clinical description, one should strive to identify reliable and precise laboratory measures to assist in diagnosis. Neuropsychological (NP) measures of cognitive dysfunction are reliable and in most cases are precise throughout disease progression. On the other hand, structural MRI (sMRI) of the brain can generate reliable and precise estimates of brain volumes. Several experimental studies have revealed abnormalities in schizophrenia using NP or sMRI measures (Andreasen et al. 1990; Ballmaier et al. 2008; Hawkins 1999; Hulshoff Pol and Kahn 2008; Leeson et al. 2008; Midorikawa et al. 2008; Segall et al. 2009; Wang et al. 2008; Watson et al. 1981). Thus, NP and sMRI are suitable candidate laboratory tests for the early identification of individuals with schizophrenia and related psychoses.

Our goal in this study was to develop and evaluate diagnostic models for recent-onset schizophrenia spectrum illnesses based on NP and sMRI variables, in combination and separately. The choice to evaluate recent-onset patients was based on our intent to develop diagnostic tools that can be of practical adjunct use in a real-life first visit to a mental health clinic. To this end, we drew participants from four geographic regions of the United States (Minnesota, New Mexico, Iowa, and Massachusetts).

Previous work by our group demonstrated the potential of applying Linear Discriminant Analysis (LDA) (Green 1978) on accurately diagnosing adult (Georgopoulos et al. 2007) and adolescent (Pardo et al. 2006) schizophrenic patients. To evaluate recent-onset patients in the current study we followed two approaches. In the first approach, based on stepwise (STP) LDA (Dixon 1992), we selected subsets of pertinent variables which would lead to faster models, risking however the rejection of useful information. In the second approach, based on Principal Component Analysis (PCA) (Green 1978) and LDA, we aimed at maximally exploiting data information from our sample, leading to potentially more accurate tests. Through these two approaches we sought to develop NP and sMRI models that could be used as biomarkers for the diagnosis of recent-onset psychotic disorders. In addition, we sought to obtain an unbiased error estimate of future performance of the modeling algorithms using external-cross-validation (Ambroise and McLachlan 2002).

Method

Subject Selection

Subjects were selected from the larger MIND Clinical Imaging Consortium (MCIC) sample (Segall et al. 2009). In the Consortium, prospective recent-onset patients and healthy subjects underwent structured CASH or SCID-I diagnostic interview evaluations by trained assessors (Andreasen et al. 1992; First et al. 2001), sMRI scanning and NP testing. The disease was labeled as of “recent-onset” when any one of the following criteria was present: (a) psychotic symptoms of any duration with a maximum lifetime exposure of 10 mg haloperidol equivalents, (b) psychotic symptoms of no more than 5 years duration, combined with no or minimal antipsychotic exposure (no more than a total lifetime exposure of the equivalent of 10 mg haloperidol), (c) psychotic symptoms of no more than 5 years duration, combined with no more than 12 weeks of antipsychotic exposure, and with no exposure during the previous 3 weeks, or (d) psychotic symptoms of no more than 2 years duration, combined with no more than 6 months of antipsychotic exposure.

Patients had one of the following diagnoses: schizophrenia, schizophreniform disorder, or schizoaffective disorder. We used data from 47 healthy controls and 28 patients (Ntotal=75) in this analysis. (From a total of 84 subjects available, 9 were excluded due to incomplete NP testing.) The demographic characteristics of the two groups are presented in Table 1 (Hollingshead and Redlich 1958). After complete description of the study to the subjects, written informed consent was obtained. Procedures were in accordance with the policies and ethical standards of the institutional review boards at the four sites and the Declaration of Helsinki (World Medical Association General Assembly 2004).

Table 1.

Subject demographics

Group Age (years)
 Duration of
Diagnosis (years)
 Treatment
Duration (years)
 Sex
 Ethnicity
 Race
 Parental SESa
 Highest Education (years)
 Handednessb
Mean SD Mean range Mean range Male Female Hispanic Non-Hispanic Native American Asian Black White 1 2 3 4 5 Mean SD R A L
Patients 24.32 5.05 2.2 0.03–12.3 0.8 0.04–4.0 22 6 5 23 0 1 4 23 1 9 13 5 0 13.67 2.23 26 1 1
Controls 26.38 6.6 NA NA NA NA 27 20 6 41 2 1 0 44 1 10 34 2 0 14.98 1.81 43 2 2
Open in a new tab
a

SES: socioeconomic status as measured by the modified Hollingshead scale. 1 = wealth, prestige; 2 = advanced degree; 3 = white collars, small businesses; 4 = semi-skilled workers, second level of education; 5 = semi-skilled workers, elementary education

b

R = right-handed; L = left-handed; A = ambidextrous

Neuropsychological Assessment

Neuropsychological assessment was based on a battery of tests (Annett 1970; Brandt 1991; Delis et al. 2001; Miller 1990; Reitan 1958; Ruff and Parker 1993; Shallice 1982; Wechsler 1997a, b; Wilkinson 1993) yielding a total of 75 scores, the specifics of which are given in Table 2. None of the NP variables was a composite of others (i.e. sums, averages), which allowed maximum non-overlapping information to be incorporated in the models without adding redundant variables. It is also important to note that none of these 75 variables was used in diagnosing patients. The approximate duration to complete these tests was 3 hours.

Table 2.

Neuropsychological variables used

NP Testa,b,c,d NP name Function/Domain assessed
1 Annett Scale of Hand Preference Continuous (SUM) Handedness
2 WRAT-3 Reading Total Reading Reading/Potential or Premorbid IQ
3 Benton Visual Retention Error Categories: Omissions Visual memory
4 Benton Visual Retention Error Categories: Distortions Visual memory
5 Benton Visual Retention Error Categories: Perseverations Visual memory
6 Benton Visual Retention Error Categories: Rotations Visual memory
7 Benton Visual Retention Error Categories: Misplacements Visual memory
8 Benton Visual Retention Error Categories: Size Errors Visual memory
9 Benton Visual Retention Error Categories: Left Errors Visual memory
10 Benton Visual Retention Error Categories: Right Errors Visual memory
11 Benton Visual Retention Number Correct Score Visual memory
12 Category Fluency Vegetables Verbal/Linguistic
13 Letter Fluency Starts With ‘F’ Verbal/Linguistic
14 Letter Fluency Starts With ‘A’ Verbal/Linguistic
15 Letter Fluency Starts With ‘S’ Verbal/Linguistic
16 Category Fluency Animals Verbal/Linguistic
17 Category Fluency Fruits Verbal/Linguistic
18 WAIS-3 Block Design Total Raw Score Spatial motor coordination
19 WAIS-3 Letter-Number Total Raw Score Working memory
20 WAIS-3 Vocabulary Total Raw Score Verbal/Linguistic
21 WAIS-3 Similarities Total Raw Score Verbal abstraction
22 WMS-3 (Immediate) Story A Recall Score Verbal declarative memory
23 WMS-3 (Immediate) Story A Thematic Score Verbal declarative memory
24 WMS-3 (Immediate) Story B 1st recall score Verbal declarative memory
25 WMS-3 (Immediate) Story B 1st recall Thematic score Verbal declarative memory
26 WMS-3 (Immediate) Story B 2nd recall score Verbal declarative memory
27 WMS-3 (Immediate) Story B 2nd recall Thematic score Verbal declarative memory
28 Trail Making Trials A: Time Problem solving/Executive
29 Trail Making Trials A: Errors Problem solving/Executive
30 Trail Making Trails B: Time Problem solving/Executive
31 Trail Making Trails B: Errors Problem solving/Executive
32 CalCap SRT Base: Reaction Time Processing speed & attention
33 CalCap CRT Base: Reaction Time Processing speed & attention
34 CalCap CRT Base: True Positive Processing speed & attention
35 CalCap CRT Base: False Positive Processing speed & attention
36 CalCap CRT SEQ1: Reaction Time Processing speed & attention
37 CalCap CRT SEQ1: True Positive Processing speed & attention
38 CalCap CRT SEQ1: False Positive Processing speed & attention
39 CalCap CRT SEQ2: Reaction Time Processing speed & attention
40 CalCap CRT SEQ2: True Positive Processing speed & attention
41 CalCap CRT SEQ2: False Positive Processing speed & attention
42 WMS-3 (Delay) Story A recall score Verbal declarative memory
43 WMS-3 (Delay) Story A Thematic score Verbal declarative memory
44 WMS-3 (Delay) Story B recall score Verbal declarative memory
45 WMS-3 (Delay) Story B Thematic score Verbal declarative memory
46 WMS-3 (Delay) Recognition Total Score Verbal declarative memory
47 WMS-3 (Delay) Percent Retention Verbal declarative memory
48 Tower of London Moves Made: 3 Ring Problem solving/Executive
49 Tower of London Moves Made: 4 Ring Problem solving/Executive
50 Tower of London Moves Made: 5 Ring Problem solving/Executive
51 Tower of London Avg. Pickup Time: 3 Ring Problem solving/Executive
52 Tower of London Avg. Pickup Time: 4 Ring Problem solving/Executive
53 Tower of London Avg. Pickup Time: 5 Ring Problem solving/Executive
54 Tower of London Avg. Total Time: 3 Ring Problem solving/Executive
55 Tower of London Avg. Total Time: 4 Ring Problem solving/Executive
56 Tower of London Avg. Total Time: 5 Ring Problem solving/Executive
57 Tower of London Total Trials Time: 3 Ring Problem solving/Executive
58 Tower of London Total Trials Time: 4 Ring Problem solving/Executive
59 Tower of London Total Trials Time: 5 Ring Problem solving/Executive
60 Tower of London Avg. Trial Time: 3 Ring Problem solving/Executive
61 Tower of London Avg. Trial Time: 4 Ring Problem solving/Executive
62 Tower of London Avg. Trial Time: 5 Ring Problem solving/Executive
63 WMS-3 Faces (Immediate) Recognition total score Visual memory
64 WMS-3 Faces (Delay) Recognition Total Score Visual memory
65 Hopkins verbal learning (Immediate) Total Raw: Trial 1 Verbal declarative memory
66 Hopkins verbal learning (Immediate) Total Raw: Trial 2 Verbal declarative memory
67 Hopkins verbal learning (Immediate) Total Raw: Trial 3 Verbal declarative memory
68 Grooved pegboard Dominant Hand: # Drops Fine motor dexterity & speed
69 Grooved pegboard Non-dominant Hand: Time Fine motor dexterity & speed
70 Grooved pegboard Non-dominant Hand: # Drops Fine motor dexterity & speed
71 Grooved pegboard Dominant Hand: Time Fine motor dexterity & speed
72 Hopkins verbal learning (Delay) Delay Total Raw Verbal declarative memory
73 Hopkins verbal learning (Delay) Recognition: Hits Verbal declarative memory
74 Hopkins verbal learning (Delay) Recognition: F/A Related Verbal declarative memory
75 Hopkins verbal learning (Delay) Recognition: F/A Unrelated Verbal declarative memory
Open in a new tab
a

WRAT: Wide Range Achievement Test—Third Edition

b

WAIS-3: Wechsler Adult Intelligence Scale—Third Edition

c

WMS-3: Wechsler Memory Scale-III

d

CalCap: California Computerized Assessment Package

sMRI Processing

Cross-site sMRI acquisition calibration and reliability were established in a preceding study using human phantoms following guidelines developed by the Biomedical Informatics Research Network (BIRN) testbed for morphometry (Jovicich et al. 2006, 2009). The MRI acquisition protocol for high-resolution coronal T1-weighted images were matched across the four different sites (University of Minnesota: Siemens 3 T Trio, University of Iowa: Siemens 1.5 T Sonata, Massachusetts General Hospital and University of New Mexico: Siemens 1.5 T Avanto): 3 T Trio: TR/ TE/flip angle=2.53 s/3.79 ms/7°, 1.5 T Sonata: TR/TE/flip angle=20 ms/6 ms/20°, 1.5 T Avanto TR/TE/flip angle 12 ms/4.76 ms/20°, FOV=16×16 cm, 128 contiguous slices (imaging matrix=256×256, in plane resolution= 0.625×0.625 mm, slice thickness=1.5 mm), NEX=3. Acquisition time for this Gradient Echo sequence was approximately 20 minutes.

Volumetric measurements of brain areas were obtained from sMRI scans in an automated manner using the Freesurfer software package (http://surfer.nmr.mgh.harvard.edu, Version 4.0.1) (Fischl et al. 2002). Minor manual intervention following the standard Freesurfer processing manuals was required for the MRI data of one subject. A total of 95 sMRI variables were used in this analysis (Table 3). The approximate total duration for an MRI acquisition was 40 minutes.

Table 3.

sMRI variables used

sMRI name sMRI name sMRI name
1 Right caudal anterior cingulate 33 Right transverse temporal 65 Left temporal pole
2 Right caudal middle frontal 34 Left caudal anterior cingulate 66 Left transverse temporal
3 Right corpus callosum 35 Left caudal middle frontal 67 Left Cerebral White Matter
4 Right cuneus 36 Left corpus callosum 68 Left Cerebral Cortex
5 Right entorhinal 37 Left cuneus 69 Left Lateral Ventricle
6 Right fusiform 38 Left entorhinal 70 Left Inferior Lateral Ventricle
7 Right inferior parietal 39 Left fusiform 71 Left Cerebellum White Matter
8 Right inferior temporal 40 Left inferior parietal 72 Left Cerebellum Cortex
9 Right isthmus cingulate 41 Left inferior temporal 73 Left Thalamus Proper
10 Right lateral occipital 42 Left isthmus cingulate 74 Left Caudate
11 Right lateral orbitofrontal 43 Left lateral occipital 75 Left Putamen
12 Right lingual 44 Left lateral orbitofrontal 76 Left Pallidum
13 Right medial orbitofrontal 45 Left lingual 77 Third Ventricle
14 Right middle temporal 46 Left medial orbitofrontal 78 Fourth Ventricle
15 Right parahippocampal 47 Left middle temporal 79 Left Hippocampus
16 Right paracentral 48 Left parahippocampal 80 Left Amygdala
17 Right pars opercularis 49 Left paracentral 81 Left Accumbens area
18 Right pars orbitalis 50 Left pars opercularis 82 Right Cerebral White Matter
19 Right pars triangularis 51 Left pars orbitalis 83 Right Cerebral Cortex
20 Right pericalcarine 52 Left pars triangularis 84 Right Lateral Ventricle
21 Right postcentral 53 Left pericalcarine 85 Right Inferior Lateral Ventricle
22 Right posterior cingulate 54 Left postcentral 86 Right Cerebellum White Matter
23 Right precentral 55 Left posterior cingulate 87 Right Cerebellum Cortex
24 Right precuneus 56 Left precentral 88 Right Thalamus Proper
25 Right rostral anterior cingulate 57 Left precuneus 89 Right Caudate
26 Right rostral middle frontal 58 Left rostral anterior cingulate 90 Right Putamen
27 Right superior frontal 59 Left rostral middle frontal 91 Right Pallidum
28 Right superior parietal 60 Left superior frontal 92 Right Hippocampus
29 Right superior temporal 61 Left superior parietal 93 Right Amygdala
30 Right supramarginal 62 Left superior temporal 94 Right Accumbens area
31 Right frontal pole 63 Left supramarginal 95 Intracranial Volume
32 Right temporal pole 64 Left frontal pole
Open in a new tab

Model Development

Two types of models were developed for discriminating recent-onset patients from healthy controls. Both analyses were based on LDA (Green 1978). The first analysis was applied on the raw data. STP-LDA was performed using Wilk’s Λ as a criterion, with an enter-criterion of p=0.05 and a remove-criterion of p=0.10. This analysis yielded classification functions from subsets of the original variables. The second analysis involved a dimensionality reduction and was applied on standardized data. For that purpose, raw values were transformed to z-scores (across subjects, within each variable) and then subjected to PCA (Green 1978). An LDA was performed using the first k principal components as predictors; k was determined using Humphrey-Ilgen’s permutation procedure (O’Connor 2000). Finally, in both analyses, prior probabilities for the two groups were set to be equal in the LDA (chance level= 50%). This allowed easier, and more accurate, data interpretation since more controls were in the study (47 vs. 28). If priors were not set equal and a model were biased towards controls, the model would have higher overall accuracy and specificity on the specific sample simply by chance (up to 47/75=62.7%), though sensitivity would be lower.

Variable Sets Used

The two analyses mentioned in the preceding paragraph were applied under three variable settings: (a) using only NP variables, (b) using only sMRI variables, and (c) using both NP and sMRI variables. This approach allowed us to evaluate how accurate the prediction is by each set of variables separately, and whether their combination might improve or worsen classification.

External Cross-Validation

In addition to developing models that discriminate the two groups, we were interested in evaluating the accuracy of our modeling algorithms on new subjects through cross-validation. For that purpose, we used an external leave-one-out loop. Specifically, before any analysis was performed, a single subject was left out and was used only to cross-validate our results, avoiding selection bias and, thus, overfitting (Ambroise and McLachlan 2002; Duda et al. 2001). This operation was iteratively applied until all subjects were left out (Fig. 1). With respect to PCA-LDA, the z-scores and principal components of each external subject were computed based on coefficients derived from the remaining subjects.

Fig. 1.

Fig. 1

Open in a new tab

Flowchart of model development and external cross-validation for the STP-LDA and the PCA-LDA

Ultimately, both STP-LDA and PCA-LDA yielded classification tables from which the performance of the two classification methods was evaluated by computing the following measures: overall accuracy of classification, sensitivity, specificity, positive likelihood-ratio, negative likelihood-ratio, χ2 and the odds-ratio (i.e. strength and direction of the association).

STP-LDA Models for All Subjects

The external cross-validation analyses described in the preceding section were aimed to assess the success (or error) rates of the classification algorithms but cannot provide consistent information concerning the relative importance of the NP and/or sMRI predictors, since those varied depending on the particular subject left out. This information can be gained by performing the same analyses using all subjects. For that purpose, we developed specific models for the STP-LDA using all 75 available subjects under the three settings described above, and evaluated the relative contribution of each variable to the respective model. Each variable’s contribution was assessed based on its standardized coefficient. We further investigated how the variables selected in the STP-LDA models relate to the disease using ANOVA with group as a fixed factor and an ANCOVA using sex, age, and intracranial volume as covariates. In the case of the PCA-LDA the analysis described above was not performed since all variables contribute to all principal components and by extension to the LDA.

Software Packages Used

PCA-LDA was performed using the MatLab software package and built-in functions, except for Humphrey-Ilgen analysis which was based on free access code by O’Connor (2000). Regarding STP-LDA this was performed using the SPSS version 15 software package. Special care was taken to remove a subject from the external leave-one-out loop prior to any analysis (Fig. 1), since the SPSS built-in stepwise function selects variables prior to removing a subject and only removes a subject prior to calculating variable coefficients. Settings for each analysis are described in the respective methods sections.

Results

The results of the external cross-validation are presented in Table 4. Ordered contributions of original variables for the proposed final STP-LDA models are presented in Table 5 and their individual relations to the disease group are presented in Table 6.

Table 4.

Results of external cross-validation

NP-only
 sMRI-only
 Combined NP-sMRI
SZ HC SZ HC SZ HC
STP-LDA Predicted SZ 20 9 18 11 18 8
Predicted HC 8 38 10 36 10 39
Sensitivity (CI)% 71.4 (51.3–86.8) 64.3 (44.1–81.4) 64.3 (44.1–81.4)
Specificity (CI)% 80.9 (66.7–90.9) 76.6 (62.0–87.7) 83.0 (69.2–92.4)
Accuracy (CI)% 77.3 (66.2–86.2) 72.0 (60.4–81.8) 76.0 (64.8–85.1)
Positive likelihood ratio (CI) 3.73 (2.10–6 .41) 2.75 (1.57–4.65) 3.78 (2.00–7.12)
Negative likelihood ratio (CI) 0.35 (0.21–0.59) 0.47 (0.29–0.73) 0.43 (0.28–0.66)
Odds ratio (CI) 10.56 (3.58–31.14) 5.89 (2.17–16.25) 8.77 (3.01–25.59)
χ2 (p) 20.22 (7× 10−6) 12.36 (4× 10−4) 17.31 (3× 10−5)
PCA-LDA Predicted SZ 22 4 19 13 25 3
Predicted HC 6 43 9 34 3 44
Sensitivity (CI)% 78.6 (59.1–91.7) 67.9 (47.7–84.1) 89.3 (71.8–97.8)
Specificity (CI)% 91.5 (79.6–97.6) 72.3 (57.4–84.4) 93.6 (82.5–98.7)
Accuracy (CI)% 86.7 (76.8–93.4) 70.7 (59.0–80.6) 92.0 (83.4–97.0)
Positive likelihood ratio (CI) 9.23 (4.16–21.55) 2.45 (1.47–3.87) 13.99 (6.02–32.00)
Negative likelihood ratio (CI) 0.23 (0.14–0.40) 0.44 (0.26–0.73) 0.11 (0.05–0.25)
Odds ratio (CI) 39.42 (10.35–149.36) 5.52 (2.02–15.10) 122.22 (23.99–622.85)
χ2 (p) 38.03 (7× 10−10) 11.6 (7× 10−4) 51.5 (7× 10−13)
Open in a new tab

CI, 95% confidence intervals for sensitivity, specificity and accuracy calculated based on the binomial theorem p, p-values

Table 5.

Variable selection and ordered contribution during STP-LDA. All-subject model

Rank NP name sMRI name NP and sMRI name
1 Benton Visual Retention: Error Categories:
   Left Errors		 Frontal: Left precentral Frontal: Left precentral
2 Grooved pegboard: Dominant Hand: Time Subcortical: Right Pallidum Tower of London: Total Trials Time: 5 Ring
3 WMS-3 (Delay): Story B recall score Subcortical: Third Ventricle WMS-3 (Delay): Recognition Total Score
4 Tower of London: Total Trials Time: 3 Ring Frontal: Right frontal pole Frontal: Left superior frontal
5 WMS-3 (Delay): Recognition Total Score Temporal: Right parahippocampal Subcortical: Left Putamen
6 Benton Visual Retention: Error Categories:
   Rotations		 Benton Visual Retention: Error Categories:
   Distortions
7 Tower of London: Total Trials Time: 5 Ring WMS-3 (Immediate): Story B 1st recall
   Thematic score
8 WMS-3 (Delay): Story B Thematic score WMS-3 (Delay): Story B recall score
9 Benton Visual Retention: Error Categories:
   Perseverations		 Subcortical: Right Accumbens area
10 Benton Visual Retention: Error Categories:
   Misplacements		 Frontal: Right superior frontal
11 Grooved pegboard: Dominant Hand: # Drops Benton Visual Retention: Error Categories:
   Misplacements
12 WAIS-3: Block Design Total Raw Score Grooved pegboard: Non-dominant Hand: Time
13 Temporal: Right temporal pole
Open in a new tab

Table 6.

Relationship to disease of individual variables from the STP-LDA

Variable name Patients vs. Controls F[1,73] p
Story B recall score Worse 79.63 3×10−13
Story B Thematic score Worse 53.22 3×10−10
Error Categories: Left Errors Worse 38.18 3× 10−8
Story B 1st recall Thematic score Worse 37.93 4× 10−8
Non-dominant Hand: Time Worse 31.54 3× 10−7
Total Trials Time: 5 Ring Worse 30.88 4× 10−7
Recognition Total Score Worse 24.39 5× 10−6
Error Categories: Distortions Worse 24.34 5× 10−6
Dominant Hand: Time Worse 22.24 10−5
Block Design Total Raw Score Worse 16.36 10−4
Error Categories: Misplacements Worse 15.99 2×10−4
Error Categories: Perseverations Worse 15.84 2×10−4
Right frontal pole Larger 11.5 0.001
Right parahippocampal Larger 8.14 0.006
Left precentral Smaller 7.56 0.008
Left Putamen Larger 6.66 0.012
Total Trials Time: 3 Ring Worse 6.03 0.016
Right Pallidum Larger 4.34 0.041
Third Ventricle Larger 4.07 0.047
Right temporal pole Larger 3.45 0.067 (ns)
Right Accumbens area Larger 3.36 0.071 (ns)
Dominant Hand: # Drops Worse 2.03 0.158 (ns)
Left superior frontal Larger 0.25 0.615 (ns)
Right superior frontal Larger 0.21 0.651 (ns)
Error Categories: Rotations Worse 0.03 0.858 (ns)
Open in a new tab

In general, models developed under PCA-LDA achieved better classification than those under STP-LDA, except for the sMRI-only setting where results were comparable. For the PCA-LDA, NP-only was better than sMRI-only models, while the combination of NP and sMRI variables provided the best classification. In contrast, classification accuracy for STP-LDA was comparable across all combinations of data (NP-only, sMRI-only, NP-sMRI).

The NP variables that were selected through STP-LDA reflected visual memory, problem solving, executive function, and spatial motor abilities. In all those NP tests patients performed worse than controls. Regarding the selected sMRI variables, they were mostly from frontal, temporal and subcortical areas. Of the sMRI variables that had a significant group effect, patients had smaller left precentral brain volume, but larger volume for all the other structures. This remained consistent even when accounting for age, sex, and intracranial volume for variables with significant differences between groups.

Discussion

Overall Diagnostic Usefulness

We applied STP-LDA and PCA-LDA under three settings (NP-only, sMRI-only, Combined NP-sMRI) to discriminate recent-onset psychotic patients and controls. Rejection of the null hypothesis that classification was due to chance was achieved for all modeling approaches under all settings (χ2≥ 11.6; p≤7×10−4). These results indicate that both methods under all three settings can contribute at a variable level to disease diagnosis (i.e. accurately identifying whether an individual does or does not suffer from a psychotic disorder).

PCA-LDA vs. STP-LDA

When comparing between the two modeling methods, there was generally better classification with the PCA-LDA than with the STP-LDA algorithm, with the exception of the sMRI-only setting where results were comparable (compare top vs. bottom of Table 4). This finding indicates that PCA-LDA is more robust as a classification model probably due to the greater amount of accounted total variance. In the case of the STP-LDA, only certain variables are selected each time and, in addition, these variables partially covary (i.e. carry common information). On the other hand, the variables selected through PCA are composites of all original variables, and are between them orthogonal (zero covariance).

NP-Only vs. sMRI-only vs. Combined NP-sMRI

Comparison of the three variable settings is slightly more complicated. When comparing individually the NP-only setting to the sMRI-only setting, better classification is achieved using only NP variables under both algorithms. This finding indicates that there is more pertinent information in the NP relative to sMRI variables and, by extension, that cognitive deficits are probably more sensitive to the occurrence of psychotic disorders than structural abnormalities. However, this finding does not necessitate that an sMRI-only model cannot be developed, but rather that the amount of information used and the approach followed did not lead to any strong sMRI diagnostic measure.

The improved classification of the NP-sMRI combination only for the PCA-LDA, but not for the STP-LDA can be explained as follows. Through PCA-LDA all variables contribute to each principal component, and thus more of the original variance is incorporated in a model. On the other hand, through STP-LDA, a selection always has to be performed and the amount of variance explained will probably not be as large. In our case, the extra variance incorporated in the PCA-LDA was relevant to discriminating healthy subjects from patients. However, if the combination of variables added information irrelevant to disease diagnosis, then classification could have been worse. It is the relevance of variables to disease diagnosis that makes the NP-sMRI PCA-LDA method the best approach.

Best Model

From the modeling methods applied, the best diagnostic outcome for all measures was provided under the Combined NP-sMRI setting of the PCA-LDA algorithm. The high values of sensitivity (89.3%), specificity (93.6%), positive likelihood-ratio (13.99), and diagnostic odds-ratio (122.22), and the low negative likelihood-ratio (0.11) suggest a diagnostic usefulness of the PCA-LDA under combined NP-sMRI. The fact that all variables need to be used in this model makes it a better candidate as an adjunct in disease diagnosis rather than a fast screening tool (i.e. approximately three and a half hours for a subject to undergo all NP and sMRI testing).

Utility of Other Models

On the other hand, when it comes to a faster model, it is more practical to use fewer variables. That can be achieved through the STP-LDA models. In these models, sensitivity and specificity are good, yet not as high as in the best model presented above. Overall, the findings on the STP-LDA models do not make them ideal screening tests, but there are indications of their usefulness as faster tools (duration of about 1 h for the NP-only) and with the introduction of more subjects in the MCIC database prediction could improve. NP-only models can be useful to a health provider since NP tests are straightforward in their administration and, even more, there are now objective computerized tests that can be widely available to the community, even in remote areas. Similarly, in those cases where patients visiting the doctor’s office have already had an MRI, an sMRI-only model could be used as an adjunct during initial evaluation.

Variable Contribution in STP-LDA

The fact that differences in NP score means were more significant than sMRI variables indicates that cognitive deficits are more sensitive for early diagnosis than volumetric structural deficits and, thus, also supports the findings discussed above. The specific variables that were selected by the STP-LDA related to memory (verbal and visual), motor dexterity and speed, visuospatial abilities, and executive function. Patients consistently performed worse in these tests relative to controls (see Table 6). These findings are in agreement with a large body of evidence on the nature of cognitive deficits in chronic and first-episode schizophrenia (Hawkins 1999; Leeson et al. 2008; Midorikawa et al. 2008; Watson et al. 1981).

The sMRI structures that were selected by STP-LDA were frontal, temporal and subcortical. The involvement of basal ganglia in schizophrenia has been reported in MRI studies of minimally medicated patients, however volumetric differences between groups have not been consistently observed, with larger, smaller, or equal volumes in patients versus controls (Ballmaier et al. 2008; Glenthoj et al. 2007; Kawasaki et al. 2004; Velakoulis et al. 2002; Wang et al. 2008). Our findings indicate a tendency for increased putamen and pallidum volumes in patients. On the other hand, the relation observed between increased ventricle size and schizophrenia is better characterized in the literature and consistently observed (Andreasen et al. 1990; Hulshoff Pol and Kahn 2008). Regarding the volumetric differences in the frontal and temporal lobes, results from previous research indicate an association between decreased volume and schizophrenia (Borgwardt et al. 2007; Hazlett et al. 2008; Kawasaki et al. 2004; Velakoulis et al. 2002). However, there are several reports for regional variability within lobes (Selemon 2001). Szeszko et al. (1999) report volume variability within lobes, specifically, more frontal subregions in the frontal lobe having increased size and posterior subregions having decreased size in patients, which agrees with our findings (increased frontal pole volume and decreased precentral volume). The only finding in our results that has not been reported in previous studies is the increased size of the right parahippocampal cortex (Baaré et al. 2001). This could be due to the fact that the patients in the study were recent-onset, and decreased volume of temporal structures is associated with increased disease duration (Velakoulis et al. 2002). On a cellular level, the volumetric differences above can be explained through synaptic pruning in the cortex (Harrison 1999; Keshavan et al. 1994; Selemon 2001) and an increase in the number of synapses in the basal ganglia (Roberts et al. 2008).

This research, which uses computerized assessment of MRI scans, a thorough neuropsychological test battery, and a sophisticated analysis procedure carries potential for clinical application. It is possible that the MRI and neuropsychological indices may be useful for cases that are difficult to diagnose; however, in the current study only research subjects who met diagnostic criteria on a standard structured interview were included in the analysis. Therefore, the potential for neuropsychological and MRI procedures to assist specifically in complex cases is not fully tested. The present findings represent a first step in identifying indices that are viable candidates for biomarkers to enhance diagnostic determinations in schizophrenia.

As this study addresses the utilization of sMRI and neuropsychological testing for classification, it is appropriate to note prior results in this sample from more traditional group-based analyses. Regarding neuropsychological testing, Sponheim et al. (2009) have reported that both the recent-onset and chronic schizophrenic subjects exhibit a wide range of cognitive deficits compared to controls, which is consistent with the body of findings documenting cognitive dysfunction in schizophrenia. Regarding sMRI, one published report describes gray matter in MCIC subjects combined with a sample of BIRN subjects in order to increase power in detecting differences between patients and controls (Segall et al. 2009). Statistically significant differences in gray matter concentration were evident between schizophrenic patients and controls for similar brain regions across the various study sites. Thus, traditional group-based analyses on sMRI and neuropsychological measures carry some diagnostic utility. Of note, however, the above studies included both recent-onset and chronic schizophrenic patients in their analyses.

Practical Considerations

It is important to keep in mind that we evaluated minimally medicated patients who resemble patients coming to the office for the first time. This makes the derived models useful adjuncts in verifying the diagnosis early in the course of the disease with high specificity and sensitivity. The use of sMRI and NP tests allows objective measures to be incorporated into disease diagnosis, thus negating the subjectivity of structured interviews while allowing, at the same time, non-specialists to contribute in screening for disease through computerized tests. This would be an asset in rural areas and primary care centers. Even more, such models can potentially be used as surrogate biomarkers for treatment response. Full description of the Mind Clinical Imaging Consortium data base and tools for data abstraction can be accessed as referenced below (see Information Sharing Statement). The databases are currently being expanded to include data on genetic variation in the sample. Analyses that examine a combination of single nucleotide polymorphisms (SNPs), structural MRI, functional MRI, and cognitive performance may further enhance the utility of biomarkers to assist with diagnostic determinations of schizophrenia.

The best model proposed is that of the PCA-LDA with all NP and sMRI variables included. However, useful information can be obtained from the NP variables only, if, for example, a faster test were desirable. In general, the accuracy of modeling algorithms is likely to be improved with increase in the number of subjects studied. Moreover, this approach holds promise for differential diagnosis, for example, between schizophrenia and bipolar disorder (Pardo et al. 2006), with and without psychotic features, as well as between schizophrenia and schizophreniform disorder. Finally, current criteria require 6 months for the specific diagnosis of schizophrenia to be established. If our models successfully pass all phases of biomarker development (Pepe 2005), they could help make these criteria more efficient and lead to early discrimination between schizophreniform disorder and schizophrenia.

Information Sharing Statement

For more information on the methods, models, applications and tools used in this article, when they are publically available, please visit https://portal.mind.unm.edu/mcic/public/. Data will be accessible on Xnat (http://www.xnat.org/) in the near future.

Acknowledgments

Supported by the United States Department of Energy under Award Number DE-FG02-99ER62764 to The Mind Research Network (formerly The Mental Illness and Neuroscience Discovery [MIND] Institute), the Department of Veterans Affairs, the American Legion Brain Sciences Chair, and the Stanley Medical Research Institute.

The authors thank Dr. Christopher Bingham for advice on the PCA, and Mary A. Jacintha and Kathleen E. Kelly for help in study coordination.

Contributor Information

Elissaios Karageorgiou, Brain Sciences Center (11B), Veterans Affairs Medical Center, One Veterans Drive, Minneapolis, MN 55417, USA karag001@umn.edu; Department of Neuroscience, University of Minnesota Medical School, Minneapolis, MN, USA; Department of Neurology, University of Minnesota Medical School, Minneapolis, MN, USA; Center for Cognitive Sciences, University of Minnesota Medical School, Minneapolis, MN, USA.

S. Charles Schulz, Department of Psychiatry, University of Minnesota Medical School, Minneapolis, MN, USA scs@umn.edu; Mind Research Network, Albuquerque, NM, USA.

Randy L. Gollub, Department of Psychiatry, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA rgollub@partners.org

Nancy C. Andreasen, Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa, IA, USA nancy-andreasen@uiowa.edu

Beng-Choon Ho, Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa, IA, USA beng-ho@uiowa.edu.

John Lauriello, Department of Psychiatry, University of New Mexico, Albuquerque, NM, USA JLauriello@salud.unm.edu.

Vince D. Calhoun, Mind Research Network, Albuquerque, NM, USA vcalhoun@unm.edu Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM, USA.

H. Jeremy Bockholt, Mind Research Network, Albuquerque, NM, USA jbockholt@mrn.org.

Scott R. Sponheim, Brain Sciences Center (11B), Veterans Affairs Medical Center, One Veterans Drive, Minneapolis, MN 55417, USA sponh001@umn.edu Department of Psychiatry, University of Minnesota Medical School, Minneapolis, MN, USA; Mental Health Patient Service Line, Veterans Affairs Medical Center, One Veterans Drive, Minneapolis, MN 55417, USA; Department of Psychology, University of Minnesota Medical School, Minneapolis, MN 55455, USA.

Apostolos P. Georgopoulos, Brain Sciences Center (11B), Veterans Affairs Medical Center, One Veterans Drive, Minneapolis, MN 55417, USA Department of Neuroscience, University of Minnesota Medical School, Minneapolis, MN, USA; Department of Neurology, University of Minnesota Medical School, Minneapolis, MN, USA; Center for Cognitive Sciences, University of Minnesota Medical School, Minneapolis, MN, USA; Department of Psychiatry, University of Minnesota Medical School, Minneapolis, MN, USA.

References

  1. Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences of the United States of America. 2002;99:6562–6566. doi: 10.1073/pnas.102102699. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Andreasen NC, Ehrhardt JC, Swayze VW, 2nd, Alliger RJ, Yuh WT, Cohen G, et al. Magnetic resonance imaging of the brain in schizophrenia. The pathophysiologic significance of structural abnormalities. Archives of General Psychiatry. 1990;47:35–44. doi: 10.1001/archpsyc.1990.01810130037006. [DOI] [PubMed] [Google Scholar]
  3. Andreasen NC, Flaum M, Arndt S. The Comprehensive Assessment of Symptoms and History (CASH). An instrument for assessing diagnosis and psychopathology. Archives of General Psychiatry. 1992;49:615–623. doi: 10.1001/archpsyc.1992.01820080023004. [DOI] [PubMed] [Google Scholar]
  4. Annett M. A classification of hand preference by association analysis. British Journal of Psychology (London, England: 1953) 1970;61:303–321. doi: 10.1111/j.2044-8295.1970.tb01248.x. [DOI] [PubMed] [Google Scholar]
  5. Baaré WF, van Oel CJ, Hulshoff Pol HE, Schnack HG, Durston S, Sitskoorn MM, et al. Volumes of brain structures in twins discordant for schizophrenia. Archives of General Psychiatry. 2001;58:33–40. doi: 10.1001/archpsyc.58.1.33. [DOI] [PubMed] [Google Scholar]
  6. Ballmaier M, Schlagenhauf F, Toga AW, Gallinat J, Koslowski M, Zoli M, et al. Regional patterns and clinical correlates of basal ganglia morphology in non-medicated schizophrenia. Schizophrenia Research. 2008;106:140–147. doi: 10.1016/j.schres.2008.08.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Borgwardt SJ, Riecher-Rossler A, Dazzan P, Chitnis X, Aston J, Drewe M, et al. Regional gray matter volume abnormalities in the at risk mental state. Biological Psychiatry. 2007;61:1148–1156. doi: 10.1016/j.biopsych.2006.08.009. [DOI] [PubMed] [Google Scholar]
  8. Brandt J. The Hopkins verbal learning test: development of a new memory test with six equivalent forms. Clinical Neuropsychologist. 1991;5:125–142. [Google Scholar]
  9. Delis DC, Kaplan E, Kramer JH. Delis-Kaplan Executive Function System (DKEFS): Examiner’s manual. San Antonio, TX: The Psychological Corporation; 2001. [Google Scholar]
  10. Dixon WJ. BMDP statistical software manual. Berkley: University of California Press; 1992. [Google Scholar]
  11. Duda RO, Hart PE, Stork DG. Pattern classification. New York: Wiley; 2001. p. 654. [Google Scholar]
  12. First MB, Spitzer RL, Gibbon M, Williams J. Structural clinical interview for DSM-IV-TR axis I disorders, research version, patient edition with psychotic screen (SCID-I/P W/PSY SCREEN) New York: New York State Psychiatric Institute, Biometrics Research; 2001. [Google Scholar]
  13. Fischl B, Salat DH, Busa E, Albert M, Dieterich M, Haselgrove C, et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron. 2002;33:341–355. doi: 10.1016/s0896-6273(02)00569-x. [DOI] [PubMed] [Google Scholar]
  14. Georgopoulos AP, Karageorgiou E, Leuthold AC, Lewis SM, Lynch JK, Alonso AA, et al. Synchronous neural interactions assessed by magnetoencephalography: a functional biomarker for brain disorders. Journal of Neural Engineering. 2007;4:349–355. doi: 10.1088/1741-2560/4/4/001. [DOI] [PubMed] [Google Scholar]
  15. Glenthoj A, Glenthoj BY, Mackeprang T, Pagsberg AK, Hemmingsen RP, Jernigan TL, et al. Basal ganglia volumes in drug-naive first-episode schizophrenia patients before and after short-term treatment with either a typical or an atypical antipsychotic drug. Psychiatry Research. 2007;154:199–208. doi: 10.1016/j.pscychresns.2006.10.002. [DOI] [PubMed] [Google Scholar]
  16. Green PE. Analyzing multivariate data. Hinsdale: The Dryden; 1978. [Google Scholar]
  17. Harrigan SM, McGorry PD, Krstev H. Does treatment delay in first-episode psychosis really matter? Psychological Medicine. 2003;33:97–110. doi: 10.1017/s003329170200675x. [DOI] [PubMed] [Google Scholar]
  18. Harrison PJ. The neuropathology of schizophrenia. A critical review of the data and their interpretation. Brain: A Journal of Neurology. 1999;122(Pt 4):593–624. doi: 10.1093/brain/122.4.593. [DOI] [PubMed] [Google Scholar]
  19. Hawkins KA. Memory deficits in patients with schizophrenia: preliminary data from the Wechsler Memory Scale-Third Edition support earlier findings. Journal of Psychiatry & Neuroscience: JPN. 1999;24:341–347. [PMC free article] [PubMed] [Google Scholar]
  20. Hazlett EA, Buchsbaum MS, Haznedar MM, Newmark R, Goldstein KE, Zelmanova Y, et al. Cortical gray and white matter volume in unmedicated schizotypal and schizophrenia patients. Schizophrenia Research. 2008;101:111–123. doi: 10.1016/j.schres.2007.12.472. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hollingshead AB, Redlich FC. Social class and mental illness; a community study. New York: Wiley; 1958. p. 442. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hulshoff Pol HE, Kahn RS. What happens after the first episode? A review of progressive brain changes in chronically ill patients with schizophrenia. Schizophrenia Bulletin. 2008;34:354–366. doi: 10.1093/schbul/sbm168. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Jovicich J, Czanner S, Greve D, Haley E, van der Kouwe A, Gollub R, et al. Reliability in multi-site structural MRI studies: effects of gradient non-linearity correction on phantom and human data. NeuroImage. 2006;30:436–443. doi: 10.1016/j.neuroimage.2005.09.046. [DOI] [PubMed] [Google Scholar]
  24. Jovicich J, Czanner S, Han X, Salat D, van der Kouwe A, Quinn B, et al. MRI-derived measurements of human subcortical, ventricular and intracranial brain volumes: reliability effects of scan sessions, acquisition sequences, data analyses, scanner upgrade, scanner vendors and field strengths. NeuroImage. 2009;46:177–192. doi: 10.1016/j.neuroimage.2009.02.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kawasaki Y, Suzuki M, Nohara S, Hagino H, Takahashi T, Matsui M, et al. Structural brain differences in patients with schizophrenia and schizotypal disorder demonstrated by voxel-based morphometry. European Archives of Psychiatry and Clinical Neuroscience. 2004;254:406–414. doi: 10.1007/s00406-004-0522-1. [DOI] [PubMed] [Google Scholar]
  26. Keshavan MS, Anderson S, Pettegrew JW. Is schizophrenia due to excessive synaptic pruning in the prefrontal cortex? The Feinberg hypothesis revisited. Journal of Psychiatric Research. 1994;28:239–265. doi: 10.1016/0022-3956(94)90009-4. [DOI] [PubMed] [Google Scholar]
  27. Leeson VC, Barnes TR, Harrison M, Matheson E, Harrison I, Mutsatsa SH, et al. The relationship between IQ, memory, executive function, and processing speed in recent-onset psychosis: 1-year stability and clinical outcome. Schizophrenia bulletin. 2008 doi: 10.1093/schbul/sbn100. [epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Midorikawa A, Hashimoto R, Noguchi H, Saitoh O, Kunugi H, Nakamura K. Impairment of motor dexterity in schizophrenia assessed by a novel finger movement test. Psychiatry Research. 2008;159:281–289. doi: 10.1016/j.psychres.2007.04.004. [DOI] [PubMed] [Google Scholar]
  29. Miller EN. California computerized assessment battery (CalCAP) manual. Los Angeles: Norland Software; 1990. [Google Scholar]
  30. O’Connor BP. SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, Inc. 2000;32:396–402. doi: 10.3758/bf03200807. [DOI] [PubMed] [Google Scholar]
  31. Pardo PJ, Georgopoulos AP, Kenny JT, Stuve TA, Findling RL, Schulz SC. Classification of adolescent psychotic disorders using linear discriminant analysis. Schizophrenia Research. 2006;87:297–306. doi: 10.1016/j.schres.2006.05.007. [DOI] [PubMed] [Google Scholar]
  32. Pepe MS. Evaluating technologies for classification and prediction in medicine. Statistics in Medicine. 2005;24:3687–3696. doi: 10.1002/sim.2431. [DOI] [PubMed] [Google Scholar]
  33. Reitan RM. Validity of the trail making test as indicator of organic brain damage. Perceptual and Motor Skills. 1958;8:271–276. [Google Scholar]
  34. Roberts RC, Roche JK, Conley RR. Differential synaptic changes in the striatum of subjects with undifferentiated versus paranoid schizophrenia. Synapse (New York, N.Y.) 2008;62:616–627. doi: 10.1002/syn.20534. [DOI] [PubMed] [Google Scholar]
  35. Robins E, Guze SB. Establishment of diagnostic validity in psychiatric illness: its application to schizophrenia. The American Journal of Psychiatry. 1970;126:983–987. doi: 10.1176/ajp.126.7.983. [DOI] [PubMed] [Google Scholar]
  36. Ruff RM, Parker SB. Gender- and age-specific changes in motor speed and eye-hand coordination in adults: normative values for the Finger Tapping and Grooved Pegboard Tests. Perceptual and Motor Skills. 1993;76:1219–1230. doi: 10.2466/pms.1993.76.3c.1219. [DOI] [PubMed] [Google Scholar]
  37. Segall JM, Turner JA, van Erp TG, White T, Bockholt HJ, Gollub RL, et al. Voxel-based morphometric multisite collaborative study on schizophrenia. Schizophrenia Bulletin. 2009;35:82–95. doi: 10.1093/schbul/sbn150. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Selemon LD. Regionally diverse cortical pathology in schizophrenia: clues to the etiology of the disease. Schizophrenia Bulletin. 2001;27:349–377. doi: 10.1093/oxfordjournals.schbul.a006881. [DOI] [PubMed] [Google Scholar]
  39. Shallice T. Specific impairments of planning. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 1982;298:199–209. doi: 10.1098/rstb.1982.0082. [DOI] [PubMed] [Google Scholar]
  40. Sponheim SR, Jung RE, Seidman LJ, Mesholam-Gately RI, Manoach DS, O’Leary DS, et al. Cognitive deficits in recent-onset and chronic schizophrenia. Journal of Psychiatric Research. 2009;44:421–428. doi: 10.1016/j.jpsychires.2009.09.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Szeszko PR, Bilder RM, Lencz T, Pollack S, Alvir JM, Ashtari M, et al. Investigation of frontal lobe subregions in first-episode schizophrenia. Psychiatry Research. 1999;90:1–15. doi: 10.1016/s0925-4927(99)00002-5. [DOI] [PubMed] [Google Scholar]
  42. Velakoulis D, Wood SJ, Smith DJ, Soulsby B, Brewer W, Leeton L, et al. Increased duration of illness is associated with reduced volume in right medial temporal/anterior cingulate grey matter in patients with chronic schizophrenia. Schizophrenia Research. 2002;57:43–49. doi: 10.1016/s0920-9964(01)00307-3. [DOI] [PubMed] [Google Scholar]
  43. Wang L, Mamah D, Harms MP, Karnik M, Price JL, Gado MH, et al. Progressive deformation of deep brain nuclei and hippocampal-amygdala formation in schizophrenia. Biological Psychiatry. 2008;64:1060–1068. doi: 10.1016/j.biopsych.2008.08.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Watson CG, Gasser B, Schaefer A, Buranen C, Wold J. Separation of brain-damaged from psychiatric patients with ability and personality measures. Journal of Clinical Psychology. 1981;37:347–353. doi: 10.1002/1097-4679(198104)37:2<347::aid-jclp2270370219>3.0.co;2-a. [DOI] [PubMed] [Google Scholar]
  45. Wechsler D. Wechsler adult intelligence scale. San Antonio: Psychological Corporation; 1997a. [Google Scholar]
  46. Wechsler D. Wechsler memory scale. San Antonio: Psychological Corporation; 1997b. [Google Scholar]
  47. Wilkinson G. Wide range achievement test. Wilmington: Wide Range, Inc; 1993. [Google Scholar]
  48. World Medical Association General Assembly. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. Journal International de Bioéthique. 2004;15:124–129. [PubMed] [Google Scholar]

RESOURCES