Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Oct 3.
Published in final edited form as: Clin Neuropsychol. 2011 Sep 9;25(6):1029–1041. doi: 10.1080/13854046.2011.585142

Challenges and opportunities for genomic developmental neuropsychology: Examples from the Penn-Drexel collaborative battery

Ruben C Gur 1, Farzin Irani 1, Sarah Seligman 1, Monica E Calkins 1, Jan Richard 1, Raquel E Gur 1
PMCID: PMC4184424  NIHMSID: NIHMS388162  PMID: 21902564

Abstract

Genomics has been revolutionizing medicine over the past decade by offering mechanistic insights into disease processes and harboring the age of “individualized medicine.” Because of the sheer number of measures generated by gene sequencing methods, genomics requires “Big Science” where large datasets on genes are analyzed in reference to electronic medical record data. This revolution has largely bypassed the behavioral neurosciences, mainly because of the paucity of behavioral data in medical records and the labor intensity of available neuropsychological assessment methods. We describe the development and implementation of an efficient neuroscience-based computerized battery, coupled with a computerized clinical assessment procedure. This assessment package has been applied to a genomic study of 10,000 children aged 8-21, of whom 1000 also undergo neuroimaging. Results from the first 3000 participants indicate sensitivity to neurodevelopmental trajectories. Sex differences were evident, with females outperforming males in memory and social cognition domains, while for spatial processing males were more accurate and faster, and they were faster on simple motor tasks. The study illustrates what will hopefully become a major component of the work of clinical and research neuropsychologists as invaluable participants in the dawning age of Big Science neuropsychological genomics.

Introduction1

At the turn of the previous century, a series of annual seminars in Niels Bohr's Institute in Copenhagen brought together leading physicists from around the world. The young physicists soon realized that the new “Knaben physick” will require a paradigm shift away from working in small isolated laboratories testing esoteric theories that few people understood, to large-scale collaborative work that could elucidate complex phenomena. Such work required engaging governments to support very expensive equipment and multiple investigators that collaborate intensely sharing plans, data and conclusions. The whole field required adjustment at almost every level, from training to modes of communications to ways of giving credit to the many investigators participating in the research efforts. The era of “Big Science” had begun, and physicists never looked back.

In the past decade the advent of genomics and the prospects of “individualized medicine” have brought Big Science to biomedical research. The genomics revolution is ongoing, and medical investigators, like their physics colleagues a century ago, are now emerging from small, highly specialized laboratories where they have worked in virtual isolation and are huddling in large groups to collaborate on studies that involve multiple national and international data collection efforts. Genomics research needs large samples and NIH and other agencies have invested resources that are unprecedented in magnitude and scope into the genome project. We have just barely begun to realize the benefits of this investment. The sequencing of the human genome has accelerated our understanding of biological functions encoded in the genome, the biological basis of diseases and the evolution and history of the human species (Lander 2011). The use of large-scale genetic studies has yielded disease genes that offer mechanistic understanding that leads directly to treatment of target populations. While the genomic approach to research has been criticized as having high costs but lacking clear hypotheses (Weinberg 2010), it is evident that any insight into relevant genomic pathways for a disorder would represent a significant theoretical advance, regardless of whether those substrates turn out to have major or minor roles in etiology. For example, genetic forms of familial hypercholesterolemia account only for a fraction of heart disease risk, yet drug interventions in related systems to lower cholesterol levels are widely used and efficacious. Genomics as a field has a general hypothesis that disease is caused by a cascade of gene-modulated processes that exert their effects through interaction with the environment. To test this hypothesis in any specific group of diseases requires large samples because gene sequencing yields multiple dependent measures hence inflating Type I error probability. But successful efforts have led to conventional hypothesis-driven research that have yielded a more complete understanding of diseases and the development of new therapies (Golub 2010). Thus, while multiple genetic abnormalities may cause a complex disorder such as schizophrenia or autism, the underlying biological mechanisms may be fewer and may reveal the pathophysiology of these disorders.

The genomics revolution, unfortunately, has largely bypassed psychiatry, behavioral neurology and clinical neuropsychology, the bio-behavioral disciplines. The main reason is that genomics of disease has advanced by crossing large databases of genotypes with medical information available on electronic medical records, which is detailed on biomarkers such as blood pressure, blood chemistry, heart rate, height and weight, but woefully lacking information pertinent to behavior. We submit that unless we are resigned to staying on the sidelines of the genomics revolution, the field of neuropsychology has to undergo a paradigm shift. The complexity of physics is matched, at the least, with the complexity of behavior, brain, and the genetic, epigenetic and environmental mechanisms affecting the brain and thereby the processes through which it regulates behavior. Single investigators studying relatively small samples with intricate methods for obtaining parameters of behavior and brain function are incapable of penetrating this complexity. The following discussion will provide an overview of the transition towards neuroscience-based computerized assessment methods in our laboratory, and how this approach was applied to the field of genomics in the context of developmental neuropsychology.

Development of the Penn Computerized Neurocognitive Battery (CNB)

Traditional paper-pencil neuropsychological tests enable us to link cognitive domains to brain regions. For instance, frontal lobe functions such as abstraction and mental flexibility can be assessed using the Wisconsin Card Sort Test (Heaton 1981) and temporal lobe functions such as memory can be tested with the Wechsler Memory Scale (Wechsler 1945). Typically, a neuropsychological battery takes half a day for a healthy person and a whole day for an individual with a brain disorder. It requires extensive training both for administration and scoring, and after the data are obtained additional time is needed for verifying accuracy of the protocol and the results, and finally data entry. Traditional paper pencil assessments have been invaluable in advancing our understanding of brain-behavior relationships. We have used it extensively in our work and found it sensitive to age effects and sex differences in neuropsychological performance in healthy men and women aged 18-45 (Figure 1). These batteries helped characterize the deficit profile in several disorders, such as schizophrenia (Saykin et al, 1991, Figure 2). This battery and others like it are being used routinely in clinical practice and research.

Figure 1.

Figure 1

Open in a new tab

Results from the traditional neuropsychological battery: Sex differences. ABF=Abstraction and Mental Flexibility; ATT= Attention; VMEM= Verbal Memory; SMEM=Spatial Memory; LAN=Language; SPA=Spatial Processing; SEN=Sensory; MOT=Motor.

Figure 2.

Figure 2

Open in a new tab

Results from the traditional neuropsychological battery: Effects of schizophrenia, before and after treatment. ABF=Abstraction and Mental Flexibility; ATT= Attention; VMEM= Verbal Memory; SMEM=Spatial Memory; LAN=Language; SPA=Spatial Processing; SEN=Sensory; MOT=Motor; NP=Neuropsychological.

Some of the shortcomings of paper-and-pencil batteries have been addressed by a recent surge in computerized batteries. For instance, several commercial as well as research based computerized batteries are available including Cogtest (http://www.cogtest.com), CogLab (http://coglab.wadsworth.com), IntegNeuro (http://www.rhistl.com/IntegNeuro.htm), Cogstate (http://www.cogstate.com), Cambridge Neuropsychological Test Automated Battery (http://www.cantab.com/camcog/default.asp), MINDSTREAMS Cognitive Health Assessment (http://www.neurotrax.com), Cognitive Drug Research (http://www.cognitivedrugresearch.com), WebNeuro (http://www.brainresource.com), Computerized Neuropsychological Test Battery (http://jgp.sagepub.com/cgi/content/abstract/4/4/211) and Computerized Multiphasic Interactive Neurocognitive Dual Display System). There are concerns regarding computerized assessments including equivalence with paper and pencil methods and applicability to computer naïve populations (Feldstein et al, 1999). Technical issues have also been raised related to operating systems, displays, mouse/keyboard sampling rates and web-based assessments that can confound accuracy of timing and stimulus presentation, although there are ways described to protect against these confounds (Cernich et al, 2007).

We have introduced a computerized neurocognitive battery composed of tests that have been used in functional neuroimaging studies, and have validated it in healthy (R. C. Gur, Ragland, Moberg, Turner et al., 2001; Gur, Richard, Hughett, Calkins et al., 2010) and patient populations (R. C. Gur, Ragland, Moberg, Bilker et al., 2001; Greenwood et al., 2007) and have also demonstrated its feasibility in elderly populations (Irani et al, in press). Here we describe the history of our transition from the traditional paper-and-pencil to the current approach to neurocognitive testing.

The introduction of functional neuroimaging as a tool for neuropsychological study has made the first dent into the traditional testing methodology. It is plainly unfeasible to conduct standard neuropsychological testing with a research participant or a patient lying supine in a scanner, sometimes with catheters attached to the arms, or deep inside a magnet bore where no metal is allowed. We discovered this already with the first study in which we measured cerebral blood flow (CBF) using the 133Xe inhalation method (Gur & Reivich 1980). Slides had to be made from the test stimuli and projected onto a small screen, and responses were either not recorded or were limited to simple button presses. With the proliferation of neuroimaging over the decades since, there has been a need to computerize paper and pencil evaluations or generate new tests in order to adapt them for an imaging environment. Traditional paper and pencil tests posed additional problems for neuroimaging. Most require complex interactions that would activate the entire cortex, and for successful mapping of brain systems test domains had to be narrowly defined and psychometrically validated as “neurobehavioral probes” for functional neuroimaging studies (Gur et al, 1992). As the brain mapping efforts continued with this methodology, more tasks were developed to probe an ever-larger set of brain systems and eventually we realized that these new tasks sampled the main behavioral domains that recruit major brain circuits. These tasks, with some necessary adjustments, could be used not only to activate specific brain circuits but also as tests to measure individual differences in performance related to these brain systems. This effort has led to the Penn Computerized Neurocognitive Battery or CNB (Figure 3).

Figure 3. The current version of the Penn Computerized Neurocognitive Battery: Examples of stimuli.

Figure 3

Open in a new tab

The CNB was initially implemented through collaboration between the University of Pennsylvania and Drexel University. Dr. Douglas Chute (Drexel) offered the PowerLaboratory© platform (Chute & Westall 1997; Gur et al, 1992) that enabled us to adapt a computerized battery composed of multiple narrowly defined tests that nonetheless sampled a wide range of neuropsychological domains (Gur et al, 2002b). Specifically, we were challenged with developing tasks that were easy enough to reduce within-scanner frustrations, but complex enough to enable measurement of individual differences. Our initial resulting battery took approximately 1½ to 2 hours to administer and was thus quicker and more efficient than our prior paper and pencil-based evaluations. The efficiency came not only from less training required but also from the automated errorless data entry and checking and the availability of both accuracy and precise speed measures. Domains assessed included abstraction and mental flexibility, attention, working memory, immediate and delayed episodic recognition memory (words, faces, shapes), verbal reasoning, spatial functioning, sensory-motor functioning and motor speed. The battery yielded scores that were validated against a traditional battery in a healthy normative sample (Gur et al, 2001b) and in a sample of patients with schizophrenia (Gur et al, 2001a; see Figure 4). The computerized platform also afforded accurate millisecond timing and image linking capabilities (Chute 1993; Westall et al, 1986).

Figure 4. The Penn Computerized Neurocognitive Battery: Results in a normative sample and in patients with schizophrenia.

Figure 4

Open in a new tab

ABF=Abstraction and Mental Flexibility; ATT= Attention; VMEM= Verbal Memory; SMEM=Spatial Memory; LAN=Language; SPA=Spatial Processing; SM=Sensory-Motor; CNT=Control; SCH=Schizophrenia patient.

Since measures of social cognition were not evaluated in the traditional battery, we developed tests of emotion recognition and discrimination based on methods described previously (Gur et al, 2002b). The stimuli were applied in functional neuroimaging studies (Gur et al, 2002a; Gur et al, 2002c; Moser et al, 2007). We adapted the facial emotion recognition task to examine individual differences and the test proved sensitive to sex differences and age effects. Deficits were evident in schizophrenia and first-degree family members, and had significant heritability (Greenwood et al, 2007; Gur et al, 2007; Habel et al, 2000; Kohler et al, 2000; Kohler et al, 2004; Kohler et al, 2003; Mathersul et al, 2009; Schneider et al, 2006; Weiss et al, 2006; Williams et al, 2009).

Over the years the battery has undergone refinements and, when Mac abandoned the classic operating system, we sadly had to part from the PowerLaboratory® platform and implement the tests in Adobe's Flash®. Our final battery included 11 neurocognitive and social cognition domains, took approximately 61.4 minutes to administer and had reasonably good psychometric properties. We also established the effects of age, education and parental education on both accuracy and speed (Gur et al, 2010).

Genomic Revolution: Computerized Assessments in Schizophrenia

The advent of the genomics revolution has led to a focus on the genetics of schizophrenia and other brain disorders. Previously, the field used a simple dichotomous model (“case-control” method) to study genetics of diseases, but more recently it was recognized that this approach would not be effective for many disorders because the disease itself could be a phenotypic construct that may not necessarily “cut at the biological joints.” For example, heart disease reflects a combination of many genes and lifestyle factors that affect biological markers such as blood pressure, cholesterol levels, obesity and other parameters that interact to determine who will express the disease.

It seems that schizophrenia, unlike some disorders such as Huntington's disease but not unlike most diseases, may also fit this model. Endophenotypes are measurable quantitative continuous phenotypes that can be directly linked to biological processes and that reflect different disease pathways (Gottesman & Gould 2003). The use of this approach to identify behavioral phenotypes, such as neuropsychological phenotypes of disorders like schizophrenia, will further increase the required sample size to thousands of people. This effort would be difficult, if not unthinkable, with the traditional assessment approach because of time, expertise, and resources for managing the paper trail. The computerized approach has allowed us to use a one-hour battery to facilitate collection of cognitive endophenotypes in large numbers of people quickly and efficiently. Training and supervision of administrators is standardized but not as time-intensive as for traditional paper and pencil batteries. As soon as testing is complete, data are automatically scored and this has provided us with data on over six thousand healthy individuals, patients with schizophrenia and their biological relatives (Almasy et al, 2008; Calkins et al, 2010; Greenwood et al, 2007; Gur et al, 2007). Overall results have indicated that first-degree biological relatives show a mean level of neuropsychological test performance that falls midway between mean performance of patients and controls in several cognitive test domains, as measured by both accuracy and speed.

Application of the Computerized Assessment Approach to Developmental Genomics

Most recently, we have set up a collaboration with Hakon Hakonarson, MD, Ph.D., a pioneer in genomic research. He was involved in collecting population-wide genotypic information in Iceland and helped to identify genes associated with several conditions such as attention deficit hyperactivity disorder (e.g., Elia et al, 2010) and others (e.g., Wang et al, in press). At the Children's Hospital of Philadelphia (CHOP), he directs the Center for Applied Genomics and collaborates with our lab on a large “Grand Opportunity” (GO) project supported by the National Institutes of Mental Health, entitled “Neurodevelopmental Genomics: Trajectories of Complex Phenotypes” (Raquel E. Gur and Hakon Hakonarson, Co-PIs). Over the past 5 years, Dr. Hakonarson's team collected genetic information (genotyped blood) from children being treated at CHOP. These children and their parents agreed to be re-contacted for future studies. Between 2006-2009, the team collected over 50,000 genetic samples and published many new discoveries. The GO grant facilitated a two-year collaboration between Penn and CHOP to collect clinical and neurocognitive phenotypic data on 10,000 genotyped children (aged 8-21), 1,000 of whom will also be recruited for neuroimaging studies. The large sample was required to generate sufficient power to detect genetic variants with a wide range of effects on variability of the measured phenotypes and to correct for multiple comparisons.

Specifically, children receive a computerized clinical assessment (GOASSESS), the computerized neurocognitive battery (CNB) and neuroimaging scans (structural magnetic resonance imaging, diffusion tensor imaging, functional magnetic resonance imaging). As part of the stimulus funding, our lab hired many people during the first year of this project to work towards this mission. We moved to a web-based system led by Jan Richard. The CNB and the clinical assessments were computerized by creating custom databases installed on laptops that enabled encrypted data to be collected offline, backed up and uploaded automatically to a server. Training for administration of the computerized battery involved a 4-hour didactic and hands-on session followed by a certification process. All scores undergo a validation process that is partially automated but may trigger involvement of a higher-level investigator. Feedback is provided to the assessors on strategies for dealing with specific situations to ensure valid and complete data collection.

Adapting the CNB for pediatric assessments required some modifications to the adult battery. Most importantly, the battery needed to be shortened even further. Therefore, identification of the minimal number of items needed to achieve reliable measures on each test was initiated. Test reduction was accomplished by a series of statistical analyses using both classical test theory and item response theory. The CNB was further adjusted to offer simplified instructions for children, ample time for practice and the option for standardized breaks every 15 minutes to manage inattention or any behavioral concerns. The stimuli for verbal subtests (e.g. Word Memory, Verbal Reasoning) were adjusted to age-appropriate levels. To motivate children and encourage their best performance, the assessor prefaced the battery with the following: “We will now do some memory and puzzle-like games on the computer. Some are easy and some are more difficult. Don't worry if you make mistakes -- everyone does. Try your hardest, working accurately and quickly. Some questions may take more time than others, and that's ok; just do your best on each one.” Children were praised for effort and overall they interfaced well with the computerized battery. Over 99% of children and adolescents tested provided valid data that was complete or mostly complete, which is consistent with data we have acquired from adults who completed the CNB.

The computerized clinical interview (GOASSESS) was developed by Monica Calkins in collaboration with Kathleen Merikangas and Marcy Burstein from the NIMH, based on a modified version of the Kiddie-Schedule for Affective Disorders and Schizophrenia (K-SADS). It is a computerized, structured interview administered by trained and certified clinical research coordinators. The psychopathology screener allows symptom and criterion-related assessment of mood disorders (depression, mania/hypomania), anxiety disorders (Overanxious/Generalized, Separation Anxiety, Specific Phobia, Social Anxiety, Panic Disorder, Agoraphobia, Obsessive Compulsive Disorder, Post-Traumatic Stress Disorder), behavioral disorders (Attention Deficit Hyperactivity Disorder, Oppositional Defiant Disorder, Conduct Disorder), Psychosis Spectrum (psychosis and prodromal symptoms), eating disorders, suicidal thinking and behavior, and treatment history. Figure 5 provides a view of the computerized clinical interview entitled GOASSESS platform. Assessors undergo a standardized training protocol that includes assigned readings, observation of live assessments, didactic instruction and supervised pair-wise peer practice. This exposure is followed by a certification process in which they conduct an interview observed by a senior assessor or clinical faculty member, obtain criterion-level performance ratings using a structured evaluation tool, and receive feedback from a certifying observer.

Figure 5. Example of the Computerized Clinical Interview (GOASSESS).

Figure 5

Open in a new tab

Data collection for this project is ongoing. At the time of this presentation, recruitment just passed 3,000 children aged 8-21. We presented some preliminary data that was promising regarding developmental trajectories. They are too preliminary to present here, but several features can be described. For example, initial results from the CNB analyses suggested the presence of sex differences in developmental trajectories. In the memory domain, for instance, while girls and boys performed similarly at younger age groups, as they grew older, girls tended to become more accurate and faster, particularly for facial memory. Females also performed more accurately and faster on social cognition tasks, while for spatial processing males were more accurate and faster, and also faster on simple motor tasks. These differences were apparent from childhood.

Neuroimaging genomics is a rapidly expanding field, and indices derived from structural and functional neuroimaging have been used as endophenotypes in large-scale studies (see (Glahn et al, 2007). The protocol applied an array of MR-based structural and functional measurement procedures, and these neuroimaging methods needed an expanded infrastructure for data acquisition, management, quality assurance and analysis. The neuroimaging protocol collects a high resolution anatomical image, measures quantitative blood flow using the Continuous Arterial Spin Labeling (CASL) paradigm, collects functional magnetic resonance imaging (fMRI) data for a working memory (fractal n-back) and social cognition task (emotion recognition) as well as resting blood oxygenation level dependent (BOLD) measures, and acquires diffusion tensor imaging (DTI) sequences. At the time of presentation, 400 people had been scanned and analyses are ongoing. These neuroimaging efforts may help bridge integrations with neuropharmacologic and genomic investigations (Gur & Gur 2010).

Conclusions

Overall, there is a new challenge presented to our field, to face a new world of rapid, efficient and large-scale computerized neuropsychological testing. We are aware that the issue is controversial, and the change causes understandable trepidation among clinicians and investigators reared on the traditional method. There is also the practical concern that perhaps our level of training will no longer be needed and we will be out of work. We believe that, to the contrary, the computerized tools will immensely enhance the demand for trained neuropsychologists who can help interpret and integrate results from biomedical studies. For the practicing clinician, availability of the computerized assessment will allow integration of individual findings within the larger context of individualized medicine practiced by our medical colleagues, and leave more time for careful clinical assessments. In addition, neuropsychologists involved in the developmental genomics revolution can help to elucidate the link between genes and trajectories for healthy brain development, relative to trajectories associated with various neurodevelopmental disorders. This can then aid in the identification of vulnerability biomarkers for early intervention efforts that can lay the foundation for accelerated progress in understanding and treating complex brain disorders. Identification of prodromal factors in at-risk individuals has been a major area of focus in psychiatric research and findings so far underscore the need for large-scale databases that integrate genomic, epigenetic, imaging and phenotypic information. The inclusion of neuropsychological test performance is an important step in the path from biology to symptoms, which can allow optimal genotype-phenotype correlations. These data are essential for science and industry to test increasingly refined hypotheses about complex disorders.

Acknowledgments

This body of work has been a collaboration among multiple groups over the years, including a steady collaboration in the Philadelphia area between the University of Pennsylvania and Drexel University. Some collaborators include psychologists who are co-authors and others from multiple disciplines including Mark Elliott, PhD, Christian Kohler, MD, James Loughead, PhD, Bruce I. Turetsky, MD, Christos Davatzikos, PhD, Hakon Hakonarson, MD, PhD, Ragini Verma, PhD and Douglas Chute, PhD. The GO grant has been funded by NIMH grant RC2MH089983.

Footnotes

1

At the Drexel University/Philadelphia Neuropsychological Society Pediatric Neuropsychology Symposium on November 5, 2010, Dr. Ruben Gur presented an overview of challenges and opportunities for genomic developmental neuropsychology, particularly as related to computerized neuropsychological assessments. This review summarizes aspects of Dr. Gur's presentation with the hope that it will stimulate thought and discussion about the future of our field

References

  1. Almasy L, Gur RC, Haack K, Cole SA, Calkins ME, Peralta JM, et al. A genome screen for quantitative trait loci influencing schizophrenia and neurocognitive phenotypes. American Journal of Psychiatry. 2008;165(9):1185–1192. doi: 10.1176/appi.ajp.2008.07121869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Calkins ME, Tepper P, Gur RC, Ragland JD, Klei L, Wiener HW, et al. Project among African-Americans to explore risks for schizophrenia (PAARTNERS): evidence for impairment and heritability of neurocognitive functioning in families of schizophrenia patients. American Journal of Psychiatry. 2010;167(4):459–472. doi: 10.1176/appi.ajp.2009.08091351. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Cernich AN, Brennana DM, Barker LM, Bleiberg J. Sources of error in computerized neuropsychological assessment. Archives of Clinical Neuropsychology. 2007;22(Supplement 1):39–48. doi: 10.1016/j.acn.2006.10.004. [DOI] [PubMed] [Google Scholar]
  4. Chute D, Westall R. Power Laboratory. Devon: MacLaboratory Incorporated; 1997. [Google Scholar]
  5. Chute DL. MacLaboratory for Psychology: Success, Failures, Economics, and Outcomes over its Decade of Development. Behavior Research Methods Instruments and Computers. 1993;25(2):180–188. [Google Scholar]
  6. Elia J, Gai X, Xie HM, Perin JC, Geiger E, Glessner JT, et al. Rare structural variants found in attention-deficit hyperactivity disorder are preferentially associated with neurodevelopmental genes. Molecular Psychiatry. 2010;15(6):637–646. doi: 10.1038/mp.2009.57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Feldstein SN, Keller FR, Portman RE, Durham RL, Klebe KJ, Davis HP. A comparison of computerized and standard versions of the Wisconsin Card Sorting Test. The Clinical Neuropsychologist. 1999;13(3):303–313. doi: 10.1076/clin.13.3.303.1744. [DOI] [PubMed] [Google Scholar]
  8. Glahn DC, Paus T, Thompson PM. Imaging genomics: mapping the influence of genetics on brain structure and function. Human Brain Mapping. 2007;28(6):461–463. doi: 10.1002/hbm.20416. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Golub T. Counterpoint: Data first. Nature. 2010;464(7289):679. doi: 10.1038/464679a. [DOI] [PubMed] [Google Scholar]
  10. Gottesman II, Gould TD. The endophenotype concept in psychiatry: etymology and strategic intentions. American Journal of Psychiatry. 2003;160(4):636–645. doi: 10.1176/appi.ajp.160.4.636. [DOI] [PubMed] [Google Scholar]
  11. Greenwood TA, Braff DL, Light GA, Cadenhead KS, Calkins ME, Dobie DJ, et al. Initial heritability analyses of endophenotypic measures for schizophrenia: the consortium on the genetics of schizophrenia. Archives of General Psychiatry. 2007;64(11):1242–1250. doi: 10.1001/archpsyc.64.11.1242. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gur R, Erwin RJ, Gur RE. Neurobehavioral probes for physiologic neuroimaging studies. Archives of General Psychiatry. 1992;49(5):409–414. doi: 10.1001/archpsyc.1992.01820050073013. [DOI] [PubMed] [Google Scholar]
  13. Gur R, McGrath C, Chan RM, Schroeder L, Turner T, Turetsky BI, et al. An fMRI study of facial emotion processing in patients with schizophrenia. American Journal of Psychiatry. 2002a;159(12):1992–1999. doi: 10.1176/appi.ajp.159.12.1992. [DOI] [PubMed] [Google Scholar]
  14. Gur R, Nimgaonkar VL, Almasy L, Calkins ME, Ragland JD, Pogue-Geile MF, et al. Neurocognitive endophenotypes in a multiplex multigenerational family study of schizophrenia. American Journal of Psychiatry. 2007;164(5):813–819. doi: 10.1176/ajp.2007.164.5.813. [DOI] [PubMed] [Google Scholar]
  15. Gur R, Richard J, Hughett P, Calkins ME, Macy L, Bilker WB, et al. A cognitive neuroscience-based computerized battery for efficient measurement of individual differences: standardization and initial construct validation. Journal of Neuroscience Methods. 2010;187(2):254–262. doi: 10.1016/j.jneumeth.2009.11.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gur R, Sara R, Hagendoorn M, Marom O, Hughett P, Macy L, et al. A method for obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies. Journal of Neuroscience Methods. 2002b;115(2):137–143. doi: 10.1016/s0165-0270(02)00006-7. [DOI] [PubMed] [Google Scholar]
  17. Gur RC, Ragland JD, Moberg PJ, Bilker WB, Kohler C, Siegel SJ, et al. Computerized neurocognitive scanning: II. The profile of schizophrenia. Neuropsychopharmacology. 2001a;25(5):777–788. doi: 10.1016/S0893-133X(01)00279-2. [DOI] [PubMed] [Google Scholar]
  18. Gur RC, Ragland JD, Moberg PJ, Turner TH, Bilker WB, Kohler C, et al. Computerized neurocognitive scanning: I. Methodology and validation in healthy people. Neuropsychopharmacology. 2001b;25(5):766–776. doi: 10.1016/S0893-133X(01)00278-0. [DOI] [PubMed] [Google Scholar]
  19. Gur RC, Reivich M. Cognitive task effects on hemispheric blood flow in humans: evidence for individual differences in hemispheric activation. Brain and Language. 1980;9(1):78–92. doi: 10.1016/0093-934x(80)90073-5. [DOI] [PubMed] [Google Scholar]
  20. Gur RC, Schroeder L, Turner T, McGrath C, Chan RM, Turetsky BI, et al. Brain activation during facial emotion processing. Neuroimage. 2002c;16(3 Pt 1):651–662. doi: 10.1006/nimg.2002.1097. [DOI] [PubMed] [Google Scholar]
  21. Gur RE, Gur RC. Functional magnetic resonance imaging in schizophrenia. Dialogues in Clinical Neuroscience. 2010;12(3):333–343. doi: 10.31887/DCNS.2010.12.3/rgur. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Habel U, Gur RC, Mandal MK, Salloum JB, Gur RE, Schneider F. Emotional processing in schizophrenia across cultures: standardized measures of discrimination and experience. Schizophrenia Research. 2000;42(1):57–66. doi: 10.1016/s0920-9964(99)00093-6. [DOI] [PubMed] [Google Scholar]
  23. Heaton R. Wisonsin Card Sorting Test Manual. Odessa: Psychological Assessment Resources Inc.; 1981. [Google Scholar]
  24. Irani F, Brensinger CM, Richard J, Calkins ME, Moberg PJ, Bilker W, et al. Computerized Neurocognitive Test Performance in Schizophrenia: A Lifespan Analysis. American Journal of Geriatric Psychiatry. doi: 10.1097/JGP.0b013e3182051a7d. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kohler CG, Bilker W, Hagendoorn M, Gur RE, Gur RC. Emotion recognition deficit in schizophrenia: association with symptomatology and cognition. Biological Psychiatry. 2000;48(2):127–136. doi: 10.1016/s0006-3223(00)00847-7. [DOI] [PubMed] [Google Scholar]
  26. Kohler CG, Turner T, Stolar NM, Bilker WB, Brensinger CM, Gur RE, et al. Differences in facial expressions of four universal emotions. Psychiatry Research. 2004;128(3):235–244. doi: 10.1016/j.psychres.2004.07.003. [DOI] [PubMed] [Google Scholar]
  27. Kohler CG, Turner TH, Bilker WB, Brensinger CM, Siegel SJ, Kanes SJ, et al. Facial emotion recognition in schizophrenia: intensity effects and error pattern. American Journal of Psychiatry. 2003;160(10):1768–1774. doi: 10.1176/appi.ajp.160.10.1768. [DOI] [PubMed] [Google Scholar]
  28. Lander ES. Initial impact of the sequencing of the human genome. Nature. 2011;470(7333):187–197. doi: 10.1038/nature09792. [DOI] [PubMed] [Google Scholar]
  29. Mathersul D, Palmer DM, Gur RC, Gur RE, Cooper N, Gordon E, et al. Explicit identification and implicit recognition of facial emotions: II. Core domains and relationships with general cognition. Journal of Clinical and Experimental Neuropsychology. 2009;31(3):278–291. doi: 10.1080/13803390802043619. [DOI] [PubMed] [Google Scholar]
  30. Moser E, Derntl B, Robinson S, Fink B, Gur RC, Grammer K. Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods. 2007;161(1):126–133. doi: 10.1016/j.jneumeth.2006.10.016. [DOI] [PubMed] [Google Scholar]
  31. Saykin AJ, Gur RC, Gur RE, Mozley PD, Mozley LH, Resnick SM, et al. Neuropsychological function in schizophrenia. Selective impairment in memory and learning. Archives of General Psychiatry. 1991;48(7):618–624. doi: 10.1001/archpsyc.1991.01810310036007. [DOI] [PubMed] [Google Scholar]
  32. Schneider F, Gur RC, Koch K, Backes V, Amunts K, Shah NJ, et al. Impairment in the specificity of emotion processing in schizophrenia. American Journal of Psychiatry. 2006;163(3):442–447. doi: 10.1176/appi.ajp.163.3.442. [DOI] [PubMed] [Google Scholar]
  33. Wang K, Diskin SJ, Zhang H, Attiyeh EF, Winter C, Hou C, et al. Integrative genomics identifies LMO1 as a neuroblastoma oncogene. Nature. 469(7329):216–220. doi: 10.1038/nature09609. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Wechsler D. A standardized memory scale for clinical use. Journal of Psychology: Interdisciplinary and Applied. 1945;19:87–95. [Google Scholar]
  35. Weinberg R. Point: Hypotheses first. Nature. 2010;464(7289):678. doi: 10.1038/464678a. [DOI] [PubMed] [Google Scholar]
  36. Weiss E, Kohler C, Nolan K, Czobor P, Volavka J, Platt M, et al. The relationship between history of violent and crimincal behavior and recognition of facial expression of emotions in men with schizophrenia and schizoaffective disorder. Aggressive Behavior. 2006;32(3):187–194. [Google Scholar]
  37. Westall R, Perkey MN, Chute DL. Accurate millisecond timing on Apple's Macintosh using Drexel's Millitimer. Behavior Research Methods Instruments and Computers. 1986;18:307–411. [Google Scholar]
  38. Williams LM, Mathersul D, Palmer DM, Gur RC, Gur RE, Gordon E. Explicit identification and implicit recognition of facial emotions: I. Age effects in males and females across 10 decades. Journal of Clinical and Experimental Neuropsychology. 2009;31(3):257–277. doi: 10.1080/13803390802255635. [DOI] [PubMed] [Google Scholar]

RESOURCES