Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Feb 24.
Published in final edited form as: J Int Neuropsychol Soc. 2010 Nov 19;17(1):7–13. doi: 10.1017/S1355617710001396

Neuropsychology 3.0: Evidence-Based Science and Practice

Robert M Bilder 1
PMCID: PMC3044645  NIHMSID: NIHMS272139  PMID: 21092355

Abstract

Neuropsychology is poised for transformations of its concepts and methods, leveraging advances in neuroimaging, the human genome project, psychometric theory, and information technologies. It is argued that a paradigm shift towards evidence-based science and practice can be enabled by innovations, including: (1) formal definition of neuropsychological concepts and tasks in cognitive ontologies; (2) creation of collaborative neuropsychological knowledgebases; and (3) design of web-based assessment methods that permit free development, large-sample implementation, and dynamic refinement of neuropsychological tests and the constructs these aim to assess. This article considers these opportunities, highlights selected obstacles, and offers suggestions for stepwise progress towards these goals.

Keywords: Neuropsychology, Genomics, Psychological Theory, Psychological Tests, Information Science, Brain

Three Generations of Neuropsychology

Neuropsychology is a relatively young discipline that already has undergone significant transformations. Without intention to offer a comprehensive historical review, it is suggested that the field has experienced two distinctive periods already, and is poised for an exciting third phase.

Neuropsychology 1.0 (1950-1979)

The idea that behavior is related to brain function has roots traced at least to Pythagoras (circa 550 BC), but systematic study of brain-behavior relations did not begin until the 19th century. Neuropsychology was recognized as a discipline distinct from applied areas of psychology or neurology only in the 1960's, as signified by first use of the term “neuropsychology” in the English biomedical literature (Klove, 1963) and establishment of the International Neuropsychological Society (1967). Early practitioners of clinical neuropsychology tended to work in neurology clinics, and focus on functional impairments associated with discrete brain lesions (reviewed in classic texts such as Heilman & Valenstein, 1993). 2 During this period, clinical assessment often relied on interpretation without extensive normative data, and many tests had significant psychometric limitations. While some neuropsychological batteries were formalized, many practitioners used tests flexibly for neuropsychological diagnosis.

Neuropsychology 2.0 (1980-present)

Widespread availability of neuroimaging mitigated the utility of clinical neuropsychology as a tool for localizing lesions and certain differential diagnostic questions, and revolutionized research for assessing brain-behavior relations, spawning the new discipline of cognitive neuroscience. The late 1970's witnessed the establishment of formal training programs in neuropsychology, then specialty board certification (1981), and ultimately the Houston Conference to codify training guidelines (Hannay, 1998). More attention was paid to classical psychometrics, with newer tests incorporating more elaborate standardization and co-norming to enable actuarial interpretation of score discrepancies. Paralleling this was an increase in forensic neuropsychology and rapid growth of symptom validity testing. Clinical neuropsychology increasingly focused on characterizing cognitive strengths and weaknesses rather than differential diagnosis, and gained traction in research on psychiatric syndromes.

Neuropsychology 3.0

This brief review suggests neuropsychology is poised for a paradigm shift leveraging its position at the interface of basic biology and clinical science, that integrates information science, and co-evolves with healthcare reform. A brief summary of contributing factors is followed by description of opportunities for transformative change.

Forces Promoting Change in Neuropsychology

Neuroimaging

Technological advances in visualizing brain structure and function already have revolutionized neuropsychology (see Neuropsychology 2.0, above), and will likely continue to do so, as we remain far from exhausting known technical limits on the spatial and temporal resolution of individual imaging modalities much less the quality of information that may be available from integration across modalities, and by applying newer neuroinformatics strategies (Van Horn et al., 2008). Already structural, diffusion, and resting state functional brain image databases are being assembled on a large scale, enabling for example the use of probabilistic atlases that enable us to apply the same kinds of actuarial approaches to quantifying brain structure that are customary in our inspection of neuropsychological test scores (Shattuck et al., 2008).

Functional neuroimaging has experienced particularly striking growth and has unique conceptual impact on neuropsychology. PubMed now contains more than 19,000 articles on “fMRI”, which is striking given that the list begins in 1988. Despite controversy about how much fMRI has advanced understanding of brain-behavior relations, there has been a clear shift to thinking about brain activation as a “dependent variable” that responds to cognitive manipulations, in contrast to the classical neuropsychology perspective focused on effects of lesions. Newer analytic strategies are revealing patterns of regional co-activation suggesting functional networks different from those identified by lesion studies and knowledge of connectional anatomy is increasing rapidly and will progress further under the aegis of the NIH Human Connectome Project (Biswal et al., 2010; Glahn et al., 2010). To leverage these advances will demand new theories of brain functional organization, and novel designs less biased by prior theories that may be wrong (Poldrack, Halchenko, & Hanson, 2009). These developments hold promise that current concepts about cognitive processes as “emergent functions” of brain activity will be supplanted by mechanistic models relating specific brain activation states to specific behavioral states, addressing empirically the “mind-brain” dilemma.

The Human Genome Project

The completion of the human genome project, dramatic cost reductions for genotyping and whole genome sequencing, and increased capacity to create transgenic models, has revolutionized virtually all areas of biomedicine. Particularly given that most well-characterized behavioral traits have heritability of approximately 50%, it is a matter of when not if we will find genomic associations for many individual differences in behavior. Recent research suggests, however, that these relations are even more complex than anticipated, and developing mechanistic models of how genetic variation yields behavioral variation will require work of unprecedented scope (Bilder, 2008), prompting a call for The Human Phenome Project to begin assembling myriad assays of phenotypic expression from molecule to mind (Freimer & Sabatti, 2003). This led to establishment of projects such as the Consortium for Neuropsychiatric Phenomics (CNP), which highlights neuropsychological function as a critical central ground to help span the chasm between molecular biology and more complex behavior (Bilder, Sabb, Cannon et al., 2009). The CNP, supported by the NIH Roadmap Initiative under the aegis of its theme “research teams of the future,” is focusing on strategies that aim to advance neuropsychiatric diagnosis beyond its current atheoretical taxonomy as expressed by the most recent edition of the Diagnostic and Statistical Manual (American Psychiatric Association, 1994), by defining neuropsychological phenotypes that possess mechanistic relations to underlying neural systems, that are important across the conventionally defined diagnostic syndromes, and that are tractable targets for basic research across species. This research agenda, which demands integration of neuropsychological science with expertise in genomics, molecular and cellular biology, systems biology, neuroimaging, psychometrics, and information sciences, is already being prioritized as part of the NIH Blueprint for Neuroscience and the NIMH Strategic Plan (see particularly the Research Domain Criteria [RDoC] initiative)(T. Insel et al., 2010; T. R. Insel & Cuthbert, 2009).

Information Science

Even if we are not on the verge of a technological “singularity” when non-biological knowledge will outstrip all biological knowledge (Kurzweil, 2005), there is little doubt that dramatic change in representation and use of human knowledge has been triggered by growth of the internet. More than two billion people use the internet (~29% of the world population) with higher usage in North America (77%), Oceania/Australia (61%) and Europe (58%)(Miniwatts Marketing Group, 2010). “Web 3.0” emphasizes more intelligent personalized search and retrieval, with “semantic web” features for structuring and efficient mining of content. Online biomedical knowledge includes PubMed with ~20 million citations and PubMed Central with ~1 million full-text articles. Bioinformatics resources include knowledgebases spanning genomics, gene expression, proteomics, molecular and cellular processes. Individual case-level genomic data are being centralized in national repositories, with phenotype data to follow. The “wisdom of crowds” was once considered an oxymoron but Wikipedia has more than 3.3 million content pages with quality comparable to the best encyclopedias (Giles, 2005). Facebook claims 500 million active users who spend 700 billion minutes per month accessing this site, and ~100,000 each month who update their Five Factor Personality scores. Despite so far limited validity data, a Google search for “brain training” yields more than 10 million hits, with some sites claiming millions of users despite subscriptions rates of $15/month. Implications of these developments for neuropsychology are vast, and include opportunities to: (1) share knowledge both within our professional community, and with the public, on a massive scale; (2) collaboratively assemble knowledge about brain and behavior; (3) engage large numbers of research participants; and (4) provide educational and clinical services in ways not previously imaginable. While systematic research and clinical application of these strategies is currently germinal, recent scholarly work (Jagaroo, 2009) and the birth of the Society for Neuroinformatics in Neuropsychology (http://www.scnn.org/) may mark the beginning of a new era.

Healthcare Revolution

Our healthcare system faces unprecedented crises, while support for research and training in neuropsychology, and the viability of neuropsychological services, are threatened by financial uncertainties afflicting governments and other institutions. Evidence based medicine (EBM) is increasingly seen as critical to providing healthcare resources in clinically and cost effective ways, and this already is impacting clinical neuropsychology, which is not always reviewed as “medically necessary.” Given widespread rapid deployment of electronic medical records in part by federal mandate, there is enormous potential to assemble relevant clinical data enabling objective evaluation of neuropsychological services alongside other diagnostic and treatment alternatives. This will be done with or without participation of specialists in neuropsychology. Beyond this is the promise of personalized medicine strategies, which will ultimately be augmented by genome-wide sequence data and personal health records including lifetime diagnostic and treatment data for every individual.

Agenda for Neuropsychology 3.0

To achieve a paradigm shift in neuropsychology capitalizing on these developments will require decades of commitment, but we possess today multiple actionable options to accelerate change and prepare for the future (see Figure 1).

Figure 1. Agenda for Neuropsychology 3.0.

Figure 1

Open in a new tab

Partially overlapping stages are suggested to advance evidence-based science and practice in neuropsychology, from ontology development, through collaborative knowledge aggregation, to web-based adaptive test development.

Formalizing Neuropsychological Concepts and Measurements

To increase shared knowledge about neuropsychology and enable its use across disciplines requires operational definition of key concepts and their inter-relations. Formal descriptions of content domains or ontologies are rapidly revolutionizing other biomedical disciplines. More than 2,000 bioinformatics resources are available now on line. Neuropsychology requires similar developments for its concepts to be represented, mined, and connected to the structure and function of underlying neural circuits, cellular systems, signaling pathways, molecular biology and genomics.

Challenges in the creation of neuropsychological ontologies include fuzzy concepts, semantic disambiguation of terms, instability and lack of consensus about concept labels. One large advantage is that abstract neuropsychological constructs are measured by objective test scores, just as latent constructs are validated with respect to observable indicators in structural equation modeling. By linking neuropsychological concepts to specific measurement methods, it is possible to define families of tests and objectively evaluate the degree to which these measure overlapping or non-overlapping constructs (for further discussion see Bilder, Sabb, Parker et al., 2009).19

Figure 2 illustrates how it is possible to begin formalizing hypotheses about complex neuropsychological concepts and the evidence that is used to support or refute these. Starting with an assertion, we can identify evidence that includes cognitive task indicators linked to specific functional processes (cognitive constructs), and measurements of brain function and structure that converge on neuroanatomic circuits. The cellular elements of this circuit model can be linked to other bioinformatics resources (including signaling pathways, molecular expression data, and gene networks; not shown).

Figure 2. Schematic Representation of a Neuropsychological Hypothesis.

Figure 2

Open in a new tab

The hypothesis includes an assertion (about “motor response inhibition”) and associated evidence. The evidence is derived from a particular publication, which used a specific cognitive task (the Stop Signal Reaction Time test) to measure a specific functional process (which in this example is the cognitive concept “response suppression” according to one author [Poldrack]). The hypothesis suggests that this process is dependent on functioning of a specific corticostriatal pathway, and this circuit (the “indirect pathway”) is linked to a graphical representation of the relevant connectional anatomy. The evidence also includes neuroimaging data, including functional MRI (fMRI) and diffusion tensor imaging (DTI) as supporting links to implicate the neuroanatomic circuit components that are putatively involved in the behavioral process.

Figure 2 also illustrates how conflicting hypotheses can be represented. For example, Poldrack and Chambers disagree about how best to describe functions of hyperdirect and indirect pathways; the model can be augmented by evidence to resolve conflicting interpretations. Further, quantitative annotation can enable automated meta-analysis. This strategy was used to estimate the heritability of “cognitive control” even though no study had assessed this directly; nevertheless it was possible to define cognitive control through other associated concepts and draw conclusions using indirect evidence (Sabb et al., 2008). Methods for meta-analytic structural equation modeling can be applied to these data, enabling tests of goodness of fit for competing hypotheses (Furlow & Beretvas, 2005; Riley, Simmonds, & Look, 2007).

No integrated resource addresses all of these issues but some relevant applications are under development. The Consortium for Neuropsychiatric Phenomics (www.phenomics.ucla.edu) includes a Hypothesis Web project offering free resources for designing multilevel graphical hypotheses, searching relevant literature, and recording qualitative and quantitative annotations particularly about cognitive concepts and measurements (see: PubGraph, PubAtlas, PubBrain, Phenomining, and Phenowiki). An affiliated project focuses specifically on cognitive concepts, cognitive tasks, and their inter-relations (see www.cognitiveatlas.org). Further development of these tools can help represent and work with neuropsychological concepts and link these to other repositories of biomedical knowledge, thereby enabling evidence-based science. Similar tools can serve evidence-based practice by formalizing hypotheses about assessment necessary to optimize differential diagnosis or select among different treatment options.

Collaborative Knowledge Building for Neuropsychology

Shared definitions of neuropsychological constructs and measurements enable systematic aggregation of neuropsychological knowledge. So far there are no large repositories for neuropsychological data despite relatively high consistency in data types and substantial homogeneity of specific variables that are collected. Neuropsychological evidence comprises primarily group data and individual case data. Group data exist primarily in research publications or proprietary manuals from test publishers. These sources are intrinsically static (once published, results do not change). Group data dissemination in clinical neuropsychology typically involves two stages: (1) a test is released by its publisher, with a manual including normative and validity data from selected clinical studies; and (2) subsequent publications describe results of studies applying the test in new samples. Updating of tests occurs only in stage 1, and a typical cycle time for revision is ten years. Test interpretation often relies solely on data from the original manual. Some users complement this with information from subsequent publications, but absent organized repositories, this is left to the initiative of the researcher or clinician. Individual case data today are mostly in private computer databases or file cabinets and not accessible outside the locations where the data were collected.

Dramatic improvements in open access to both group and individual case data are feasible using existing technology. The neuropsychology community can immediately assemble databases that summarize results of published studies. Just as meta-analytic results are compiled by authors of systematic review papers, we can collaboratively assemble published data about specific tests for online access. An example can be found at www.neuropsychnorms.com, which enables users to input individual test scores and receive immediate reports comparing these to published findings (Mitrushina, Boone, Razani, & D'Elia, 2005). With community engagement the scope of this work could be expanded greatly, probably covering most relevant published literature within a few years. Since many papers include healthy groups, meta-analytic normative databases could rapidly rival many standardization samples, and accrued data on new clinical samples, treatment effects, and predictive validity could grow dynamically – as fast as the studies are published.

It is assumed that individual case data can never be released without careful consideration of informed consent and privacy protections; these issues are extremely important and complex, but because there is insufficient space to elaborate these issues are not discussed further here. The primary sources of individual case data are the original test publishers, independent researchers, and clinics. Publishers tend to maintain individual case data as proprietary but release such data under certain circumstances. Researchers tend to keep data secure at least until they have published findings, and often longer, but might release data if there were a national repository that appropriately credited contributions.

An exciting possibility is that clinics and clinicians could contribute data from every examined patient in real time. If this were done, the clinical validity data for major neuropsychological tests would grow very rapidly, and provide opportunities to compare any individual patient examined to customized reference groups stratified by demographic characteristics, or by scores on other cognitive tests. Users could be provided tools to effectively filter on diagnostic characteristics, and given that there might be variability in the credibility of different sources, users could further filter on the characteristics of the clinics providing data. A national bank for neuropsychological data could revolutionize both research and assessment practices, enabling rapid aggregation of information regarding under-studied populations and that can support evidence-based effectiveness studies that will be critical for research and public healthcare decision making.

If individual case data are assembled at the item level it will be possible to analyze data using modern psychometric theory, leading to new and improved assessment methods. Community consortia could conduct not-for-profit normative and validation studies. Assuming there are ~5,000 neuropsychologists in the United States (based on memberships in INS, APA Division 40, The American Academy of Clinical Neuropsychology, and the National Academy of Neuropsychology) it is exciting to imagine progress that could be made if each examined even one person per year as part of a national consortium.

Assessment innovation

The most widely used assessment strategies in neuropsychology have undergone little fundamental change over the last century, despite breakthroughs in cognitive neuroscience, neuroimaging, psychometric theory, and human-machine interfaces. Test revisions using traditional print publishing can also have unintended consequences. For example, the WAIS-IV/WMS-IV revisions have been criticized for failing to consider back-compatibility issues that may invalidate clinical interpretations (Loring & Bauer, 2010). Promising experimental paradigms typically languish for decades in the lab prior to use in clinics. Meanwhile, web-based acquisition strategies enable rapid collection of data from widely distributed populations using adaptive testing strategies likely to at least double efficiency in construct measurement, and when constructs are correlated (as is true of most cognitive constructs), efficiency gains may be higher. One study found a 95% average reduction in items administered using a computerized adaptive test relative to administering all items on the original scales (Gibbons et al., 2008). Further, use of modern psychometric theory enables preservation of robust back-compatibility with prior test versions, simultaneously enabling introduction of new content and new constructs after these are validated (addressing the primary critique of Loring & Bauer, 2010).

Neuropsychological test development can move forward rapidly if we embrace modern technology, adopt modern psychometric theory, and collaborate. First, neuropsychology needs to embrace computerized assessment. Some express fear that computer tests will somehow replace clinicians, or miss important observations. But the computer is just a tool enabling presentation of certain stimuli and collection of certain responses, and properly used can clearly outperform a human examiner in precision and rapid implementation of adaptive algorithms. One clear advantage of computer timing precision is that it enables implementation of methods from cognitive neuroscience that rely on more subtle task manipulations and trial-by-trial analyses, which can be more sensitive and specific to individual differences in neural system function. To the extent that future computer logic may provide prompts for differential diagnosis, test selection, or test interpretation, this would only supplement and enhance clinical decision-making.

A second bolder step will involve web assessment. This idea often triggers the same anxieties raised about computerized assessment, plus concerns that examiners cannot adequately control conditions of testing, be confident that test-takers are performing tasks as instructed, or even be sure about the identities of test-takers. There are further concerns about individual differences in computer literacy, and the “digital divide” that prevents equal access to the internet. The first class of problems has technological solutions including embedded validity indicators, on-line video surveillance, and anthropometric identifiers. But elaborate surveillance strategies are not necessary for some research and even select clinical applications. There are many people who will try their best, will follow instructions, and will generate valid results, without such interventions. This is a particularly important point for psychometric test development and specific research questions, particularly genetic studies that require large samples. In contrast to conventional test development efforts involving hundreds of participants over years, web-based protocols can acquire hundreds of thousands of participants in months. Given algorithms for item-level response monitoring and automated consistency checks, there is much greater opportunity than in most current tests to detect outlying response patterns of uncertain validity. Because “brain testing” and “brain training” applications are already proliferating without quality control, there is a pressing need for neuropsychologists to participate, establish guidelines, and assure the responsible use of such applications.

Soon many individuals will be completing web-based tests of brain function in the privacy of their homes using a wide range of web-enabled devices. Rich longitudinal behavioral data will be stored in repositories, along with electronic medical records, complete genome sequences, and automatically aggregated information about environmental exposures based on individual life history. Clinicians will need to develop competencies in the use of data mining tools to effectively manage and interpret torrents of information. The neuropsychologist of the future will synthesize these data and then determine what needs to be done in lab, office or clinic, and how to direct patients towards optimal therapeutic options.

Overcoming Obstacles

These rosy visions of the future depend on multiple changes, some of which are fundamental to both neuropsychological research and clinical practice. The most critical current bottle-neck is achieving consensus frameworks for describing neuropsychological concepts and their measurement. Agreement on terms may seem difficult but there exist already platforms to achieve this aim (see www.cognitiveatlas.org), and engagement in such collaborative efforts may be an achievable goal for neuropsychological membership organizations. Even after we agree on terms, we will still face obstacles in knowledge aggregation, because existing data vary widely in the ways these are currently maintained, and in the quality with which these were originally acquired. In the longer term, it is likely that publication of research findings will be increasingly structured and data will be “deposited” in a case-wise fashion, fostering capacity for group analysis but raising additional challenges and possibly threats to academic innovation (i.e., will scientists be supported to pursue directions that deviate markedly from “standardized” data frameworks?). In the shorter term, there are opportunities for aggregation of clinical data, but standards for quality control need to be developed, implemented, and monitored. But this aggregation of an adequate knowledge-base is critical to foster acceptance of new methods for assessment, because the responsible researcher or clinician understandably desires to use the best validated methods available. This final stage – development of novel methods – may appear the most daunting but is facilitated by rapid development of relevant technologies, and indeed a true revolution in current assessment methods is achievable with existing technology. The greater obstacles may be financial, given that current funding for test development depends largely on a relatively small “niche” print publishing market. To overcome this, it may be that we need to encourage broader public interest in brain function, while simultaneously developing frameworks to assure the responsible deployment of methods that are being widely disseminated.

In summary, dramatic changes in science, technology, and society now offer us great opportunities and grand challenges to advance our shared mission as neuropsychologists; it is hoped that by working collaboratively Neuropsychology 3.0 will be seen as a ground-breaking success in biomedicine, and pave the road to Neuropsychology 4.0.

Acknowledgements

Dr. Bilder has received consulting fees or honoraria from Roche, CHDI, Johnson & Johnson, Pfizer, Cypress Pharmaceutical, and Dainippon Sumitomo within the last 12 months and is a shareholder in Cogtest Inc, part of the Cognition Group; none of these relationships poses conflicts of interest with the material presented in this article. This work was supported by the Consortium for Neuropsychiatric Phenomics (NIH Roadmap for Medical Research grants UL1-DE019580, RL1LM009833), the Cognitive Atlas project (R01MH082795), and the Michael E. Tennenbaum Family Center for the Biology of Creativity.

Literature Cited

  1. American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) American Psychiatric Association; Washington, DC: 1994. [Google Scholar]
  2. Bilder RM. Phenomics: building scaffolds for biological hypotheses in the post-genomic era. Biol Psychiatry. 2008;63(5):439–440. doi: 10.1016/j.biopsych.2007.11.013. doi: S0006-3223(07)01140-7 [pii] 0.1016/j.biopsych.2007.11.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bilder RM, Sabb FW, Cannon TD, London ED, Jentsch JD, Parker DS. Phenomics: the systematic study of phenotypes on a genome-wide scale. Neuroscience. 2009;164(1):30–42. doi: 10.1016/j.neuroscience.2009.01.027. doi: S0306-4522(09)00048-7 [pii] 10.1016/j.neuroscience.2009.01.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bilder RM, Sabb FW, Parker DS, Kalar D, Chu WW, Fox J. Cognitive ontologies for neuropsychiatric phenomics research. Cogn Neuropsychiatry. 2009;14(4-5):419–450. doi: 10.1080/13546800902787180. doi: 913383678 [pii] 10.1080/13546800902787180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Biswal BB, Mennes M, Zuo XN, Gohel S, Kelly C, Smith SM. Toward discovery science of human brain function. Proc Natl Acad Sci U S A. 2010;107(10):4734–4739. doi: 10.1073/pnas.0911855107. doi: 0911855107 [pii] 10.1073/pnas.0911855107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Freimer N, Sabatti C. The human phenome project. Nat.Genet. 2003;34(1):15–21. doi: 10.1038/ng0503-15. [DOI] [PubMed] [Google Scholar]
  7. Furlow CF, Beretvas SN. Meta-analytic methods of pooling correlation matrices for structural equation modeling under different patterns of missing data. Psychol Methods. 2005;10(2):227–254. doi: 10.1037/1082-989X.10.2.227. doi: 2005-07009-006 [pii] 10.1037/1082-989X.10.2.227. [DOI] [PubMed] [Google Scholar]
  8. Gibbons RD, Weiss DJ, Kupfer DJ, Frank E, Fagiolini A, Grochocinski VJ. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatr Serv. 2008;59(4):361–368. doi: 10.1176/appi.ps.59.4.361. doi: 59/4/361 [pii] 10.1176/appi.ps.59.4.361. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Giles J. Internet encyclopaedias go head to head. Nature. 2005;438(7070):900–901. doi: 10.1038/438900a. doi: 438900a [pii] 10.1038/438900a. [DOI] [PubMed] [Google Scholar]
  10. Glahn DC, Winkler AM, Kochunov P, Almasy L, Duggirala R, Carless MA. Genetic control over the resting brain. Proc Natl Acad Sci U S A. 2010;107(3):1223–1228. doi: 10.1073/pnas.0909969107. doi: 0909969107 [pii] 10.1073/pnas.0909969107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Hannay HJ. Proceedings of the Houston Conference on Specialty Education and Training in Clinical Neuropsychology, September 3-7, 1997, University of Houston Hilton and Conference Center. Archives of Clinical Neuropsychology. 1998;13(2):157–250. [Google Scholar]
  12. Heilman KM, Valenstein E. Clinical Neuropsychology. Vol. 3. Oxford University Press; New York: 1993. [Google Scholar]
  13. Insel T, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167(7):748–751. doi: 10.1176/appi.ajp.2010.09091379. doi: 167/7/748 [pii] 10.1176/appi.ajp.2010.09091379. [DOI] [PubMed] [Google Scholar]
  14. Insel TR, Cuthbert BN. Endophenotypes: bridging genomic complexity and disorder heterogeneity. Biol Psychiatry. 2009;66(11):988–989. doi: 10.1016/j.biopsych.2009.10.008. doi: S0006-3223(09)01208-6 [pii] 10.1016/j.biopsych.2009.10.008. [DOI] [PubMed] [Google Scholar]
  15. Jagaroo V. Obstacles and Aids to Neuroinformatics in Neuropsychology. Neuroinformatics for Neuropsychology. 2009:85–93. [Google Scholar]
  16. Klove H. Clinical Neuropsychology. Med Clin North Am. 1963;47:1647–1658. [PubMed] [Google Scholar]
  17. Kurzweil R. The Singularity is Near: When Humans Transcend Biology. Viking; New York: 2005. [Google Scholar]
  18. Loring DW, Bauer RM. Testing the limits: cautions and concerns regarding the new Wechsler IQ and Memory scales. Neurology. 2010;74(8):685–690. doi: 10.1212/WNL.0b013e3181d0cd12. doi: 74/8/685 [pii] 10.1212/WNL.0b013e3181d0cd12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Miniwatts Marketing Group Internet World Stats Usage and Population Statistics. 2010 Retrieved 8/11/2010, 2010, from http://www.internetworldstats.com/
  20. Mitrushina MN, Boone KB, Razani J, D'Elia LF. Handbook of normative data for neuropsychological assessment. Oxford University Press; USA: 2005. [Google Scholar]
  21. Poldrack RA, Halchenko YO, Hanson SJ. Decoding the large-scale structure of brain function by classifying mental States across individuals. Psychol Sci. 2009;20(11):1364–1372. doi: 10.1111/j.1467-9280.2009.02460.x. doi: PSCI2460 [pii] 10.1111/j.1467-9280.2009.02460.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Riley RD, Simmonds MC, Look MP. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol. 2007;60(5):431–439. doi: 10.1016/j.jclinepi.2006.09.009. doi: S0895-4356(06)00403-3 [pii] 10.1016/j.jclinepi.2006.09.009. [DOI] [PubMed] [Google Scholar]
  23. Sabb FW, Bearden CE, Glahn DC, Parker DS, Freimer N, Bilder RM. A collaborative knowledge base for cognitive phenomics. Mol Psychiatry. 2008;13(4):350–360. doi: 10.1038/sj.mp.4002124. doi: 4002124 [pii] 10.1038/sj.mp.4002124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Shattuck DW, Mirza M, Adisetiyo V, Hojatkashani C, Salamon G, Narr KL. Construction of a 3D probabilistic atlas of human cortical structures. Neuroimage. 2008;39(3):1064–1080. doi: 10.1016/j.neuroimage.2007.09.031. doi: S1053-8119(07)00809-9 [pii] 10.1016/j.neuroimage.2007.09.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Van Horn JD, Bandettini PA, Cheng K, Egan GF, Stenger VA, Strother S. New Horizons for the Next Era of Human Brain Imaging, Cognitive, and Behavioral Research: Pacific Rim Interactivity. Brain Imaging Behav. 2008;2(4):227–231. doi: 10.1007/s11682-008-9045-0. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES