A cognitive neuroscience based computerized battery for efficient measurement of individual differences: Standardization and initial construct validation

Abstract

There is increased need for efficient computerized methods to collect reliable data on a range of cognitive domains that can be linked to specific brain systems. Such need arises in functional neuroimaging studies, where individual differences in cognitive performance are variables of interest or serve as confounds. In genetic studies of complex behavior, which require particularly large samples, such trait measures can serve as endophenotypes. Traditional neuropsychological tests, based on clinical pathological correlations, are protracted, require extensive training in administration and scoring, and leave lengthy paper trails (double-entry for analysis). We present a computerized battery that takes an average of 1 hour and provides measures of accuracy and speed on 9 neurocognitive domains. They are cognitive neuroscience-based in that have been linked experimentally to specific brain systems with functional neuroimaging studies. We describe the process of translating tasks used in functional neuroimaging to tests for assessing individual differences. Data are presented on each test with samples ranging from 139 (81 female) to 536 (311 female) of carefully screened healthy individuals ranging in age from 18 to 84. Item consistency was established with acceptable to high Cronbach alpha coefficients. Inter-item correlations were moderate to high within domain and low to nil across domains, indicating construct validity. Initial criterion validity was demonstrated by sensitivity to sex differences and the effects of age, education and parental education. These results encourage the use of this battery in studies needing an efficient assessment of major neurocognitive domains such as multisite genetic studies and clinical trials.

1. Introduction

There is increased demand for an efficient and reliable method of measuring individual differences in cognitive domains that can be linked to brain systems. The application of functional neuroimaging with “neurobehavioral probes” (Gur et al., 1992) has contributed to understanding complex measures by dissociating their more basic components. Efforts to integrate neurobiology and genetics in studying heritable brain disorders increasingly incorporate quantitative continuous cognitive measures. Such “endophenotypes” complement the dichotomous diagnostic approach applied in genetic studies, and are needed to construct a mechanistic neurobiological model of neurodevelopmental disorders such as schizophrenia and autism (Gottesman and Gould, 2003, Gur et al., 2007), and in other disorders affecting performance.

The increased demand for efficient neurocognitive testing has confronted limitations of available test batteries, which are clinically based paper-and-pencil tests requiring extensive training in administration and scoring and producing a paper trail unmanageable in large-scale studies. Furthermore, traditional neuropsychological batteries are based on clinical-pathological correlations, where patients with brain disorders are tested and deficits are linked to disease-related brain systems. The advent of functional neuroimaging has enabled experimental approaches to isolating brain systems recruited for specific behavioral tasks. The evolving experimental fields of cognitive and affective neurosciences (e.g., Davidson, Jackson and Kalin, 2000; Posner and DiGirolamo, 2000; Panksepp, 1998) have yielded novel insights with more narrowly defined behavioral tasks. There has therefore been demand to adapt such “cognitive neuroscience-based” tasks into computerized tests that have been linked to specific brain systems in functional neuroimaging studies (e.g., RFA-MH-08-090. Adapting Basic Cognitive Measures for Clinical Assessment of Schizophrenia see http://www.nimh.nih.gov/research-funding/grants/requests-for-applications.shtml).

The process of translating concepts and tasks developed by basic neuroscientists into clinically applicable instruments is challenging beyond the heterogeneity of clinical manifestations and the extent and pervasiveness of deficits. Several issues require consideration when adapting functional neuroimaging tasks for use as neurocognitive tests to determine individual differences. First, there is an inherent contrast between the goals of tasks in functional neuroimaging compared to tests used for establishing individual differences. The goal of tasks in functional imaging studies is to activate a specific neural circuitry, and the task is made deliberately easy so that performance is nearly perfect to avoid frustration confounds, or performance is used as a “nuisance variable.” Even when comparing patients to controls on a task where patients are putatively impaired, such methodological considerations have motivated investigators to seek ways for equating difficulty levels when contrasting activation between groups, at the expense of making the resulting tasks different (Holcomb et al., 2000). In contrast to tasks for neuroimaging studies, tests used in psychometric trait assessment focus on individual differences and must avoid ceiling and floor effects. Such tests need to walk the tightrope between being too easy, which will mask individual differences, and too hard, which can decrease motivation. It is especially important to construct tests so that patients have some experience of success and are not totally frustrated, while healthy people are sufficiently challenged. Finally, it is necessary to consider that some clever but subtle manipulations that yield elegant results in college undergraduates may prove of limited utility in patients with brain disorders.

We have developed and described a systematic procedure (Gur et al., 1992) for selecting appropriate behavioral constructs, assembling test items, and performing the various stages of validation. We have developed a battery of computerized tests and established its comparability with a traditional neuropsychological battery in a healthy normative sample (Gur et al., 2001a) and in patients with schizophrenia (Gur et al., 2001b). This battery has been applied in large-scale genetic studies yielding measures with substantial heritability and linkage (Aliyu et al., 2006; Greenwood et al., 2007; Gur et al., 2007a,b; Almasy et al., 2008). Here we present the method for developing the computerized battery, an evaluation of its psychometric properties, and initial evidence for its construct validity.

The computerized neurocognitive battery (CNB) takes approximately an hour to administer and includes tests that measure the following domains: abstraction and mental flexibility, attention, working memory, episodic memory (word, face, and spatial recognition memory), language reasoning, spatial processing, sensorimotor, motor speed, and emotion identification. The CNB yields measures of accuracy (number of correct responses) and speed [median response time (RT) for correct items]. This feature permits evaluation of the individual differences in strategy pertinent to the speed-accuracy tradeoff (Smith and Kounios, 1996). The previous version of the CNB has been supplemented with measures of working memory (the letter n-back paradigm, extensively validated in functional neuroimaging studies, e.g., Braver et al., 1997; Ragland et al., 1997; 2002; Rodriguez-Jimenez et al., 2009; Minzenberg et al., 2009), and a motor speed test (finger tapping, another measure of a narrow domain extensively validated with functional neuroimaging, e.g., Aizenstein et al., 2004). The current version also replaced the facial emotion identification test that only uses black-and-white happy and sad Caucasian faces. The new emotion identification test is based on a more advanced method used for face affect acquisition, described earlier in this journal (Gur et al., 2002a). The test includes four emotions (happy, sad, anger and fear), and uses an ethnically diverse set of posers. Here we present the CNB method of construction, normative data, and construct validation data by examining inter-correlations among the tests, and initial criterion validation through examining its sensitivity to performance-related individual differences dimensions including sex differences and age and education effects.

2. Method

2.1. Validation sample

Samples used during stages of developing and validating the battery included primarily college undergraduates who received course credit for participation. The sample for the normative study consisted of healthy volunteers recruited by a consortium of research centers. The sample size varies from 139 to 539 (see Table 1) depending on their inclusion in the participating sites. Participants were recruited through advertising in community outlets including newspapers, shopping centers, and community centers and organizations. Potential participants were screened medically and psychiatrically as detailed elsewhere (Calkins et al., 2006; Aliyu et al., 2006; Gur et al., 2007a) and only individuals with no medical or psychiatric diagnosis were included in this sample. In addition, participants had no first-degree relatives with psychotic or mood disorders.

Table 1.

Sample sizes (#F=number of females), administration times, means, standard deviations (SD) and Cronbach’s alpha coefficients for the tests in the battery.

			Admin. Time		ACCURACY			SPEED
DOMAIN	TEST	N (#F)	Mean1	SD	Mean2	SD	Alpha	Mean3	SD	Alpha
ABF (Abstraction/flexibility)	PCET	490 (285)	6.2	4.1	1.7	0.8	*	2542.8	1486.8	*
ATT (Attention)	CPT N	**	**	**	57.7	5.2	0.924	474.1	71.5	0.969
	CPT L	**	**	**	55.8	5.7	0.931	501.7	67.5	0.955
	CPT	139 (81)	8.6	0.8	113.5	8.8	0.954	486.5	67.7	0.977
WM (Working memory)	LNB	367 (217)	11.4	1.0	27.9	2.7	0.770	544.7	137.5	0.816
MEM (Episodic memory)	WORD IMM	392 (218)	3.8	1.0	35.1	4.7	0.713	1261.4	358.1	0.904
	WORD DEL	390 (216)	1.7	1.0	34.5	3.7	0.726	1264.2	392.5	0.876
	FACE IMM	536 (311)	2.4	0.7	32.1	4.0	0.698	1883.3	674.0	0.908
	FACE DEL	531 (309)	2.1	0.6	33.7	4.2	0.746	1590.9	574.1	0.872
	SPATIAL IMM	483 (281)	2.3	0.6	15.6	2.5	0.553	1559.0	526.3	0.814
	SPATIAL DEL	479 (279)	1.0	0.4	15.3	2.6	0.558	1401.4	421.2	0.777
LAN (Language)	PVRT	213 (119)	5.0	2.5	65.6	20.9	0.873	9366.7	4182.8	0.901
SPA (Spatial)	JOLO	539 (313)	5.5	1.9	22.0	5.9	0.898	2976.6	1184.9	0.937
SM (Sensorimotor)	MP	506 (287)	1.8	1.7	***	***	***	780.7	314.1	0.915
MOT (Motor speed)	TAP	394 (225)	6.8	1.1	***	***	***	108.5	14.6	0.963
EMO (Emotion identification)	ER40	448 (226)	2.8	0.8	33.6	3.1	0.587	2118.6	693.5	0.919

¹minutes

²#Correct

³milliseconds

^*Not amenable for calculating

^**Data available for entire test

^***No errors

2.2. The development of CNB

We have developed a set of computerized neurobehavioral measures using tests that incorporate behavioral neuroscience-based tasks used in functional neuroimaging. Our approach to the task-to-test adaptation and the validation process was detailed in Gur et al. (1992). Briefly, we constructed tasks that can be presented with a wide range of difficulty levels while maintaining dimensional integrity and comparability between the “offline” assessment of individual differences and the system activated in a functional neuroimaging study. Tasks first undergo a process of construction and evaluation by a team of experimental and clinical investigators. This group helps carry the tasks through conceptualization, task assembly, stimulus assessment, and constructing a preliminary version. The resulting version is submitted to psychometric study to assess reliability and construct validity and obtain normative data. Data are also obtained at this stage to document comparability of test versions and the effects of retesting (“practice effects”). Tests included in the CNB have undergone this development process. The “pipeline” for adapting neuroscience based tasks for use as psychometric tests is represented in Figure 1.

Figure 1.

A flow diagram for adapting cognitive neuroscience tasks as psychometric tests in neuroimaging and clinical studies.

The following stages are undertaken: 1. Conceptualization. This takes place in weekly meeting of faculty, fellows and students where neuroscience literature is reviewed with focus on its relevance to neurodevelopmental disorders. At this stage we consult with basic science collaborators and colleagues and often invite consultants from other institutions. 2. Assembly. Potential stimuli are examined by the group and discussed in relation to complexity, physical characteristics, and suitability for patients. At this phase we also select the response modality and assemble the first version of a test, which includes a large number of items. This stage can require a large-scale project, as when we recorded facial expression from 150 actors and actresses to generate the stimuli for the facial emotion identification tests (Gur et al., 2002a). Initial test versions are first evaluated internally by members of the center and feedback is obtained. Based on this experience, instructions are evaluated and revised. We involve experts in education and linguistics for feedback on content and presentation of the instructions. We then assemble an initial computerized version of the test that includes a training module. 3. Screening. This version is then piloted with students in a computer laboratory setting, where about 100–150 participants are tested in groups averaging 10 people. They take the tests under supervision of laboratory staff. Here we emphasize conscientious performance by test takers and obtaining thorough feedback from participants. 4. Construction. In this phase we perform an item analysis of the pilot data and, based on this analysis, we assemble alternate forms of the test. We then administer the test to a small sample of a targeted patient sample and community controls (N= ~20 in each group), and perform item analysis to arrive at the final version that is submitted to a more rigorous validation process. Based on this experience and the psychometric analysis of the pilot data, we finalize the test. This includes instructions, training procedures, scoring and quality assurance. 5. Validation. The newly developed test is administered in conjunction with the current version of the CNB to a normative sample of healthy people from the community. Here we examine correlations between test performance and demographic features. 6. Dissemination. When a test is validated we make it available to the scientific community. The current versions of the tests have been written in the Adobe Flash® platform and are available via the web. The web-based platform provides a selection of computerized batteries as well as automated scoring and other features. An investigator interested in using the CNB can register for an account on the PennCNP® website at https://penncnp.med.upenn.edu/request.pl by submitting a form to verify research credentials and IRB compliance.

2.3. Method of administration

The CNB is administered using clickable icons on desktop or laptop computers, in a fixed order. The tests were implemented on Macintosh® computers using the PowerLaboratory® program (Chute and Westall, 1997). An Applescript® routine is used to collect participant IDs and basic demographic information and to present the tests in a prescribed order. A research assistant reads instructions for each test and observes as the participant performs the tests. Assistants are rained to administer the tests in a standard fashion fostering optimal performance without aiding the participant. The test administrator provides a status code and comments regarding the validity of collected data. The results are uploaded to a data repository using an automated script, and scored using a program written in the Python programming language. A neuropsychologist reviews test sessions to determine data validity. For each domain, accuracy (number of correct responses) and speed (response time for correct answers) are computed.

As a first step in testing, the subject is acquainted with the computer and mouse and perform an un-speeded version of the Mouse Practice task ((MP; Gur et al., 2001a). This practice is administered at the beginning of the session to assure the participant has sufficient skills in moving and clicking the mouse. Such practice helps “level the playing field” for people inexperienced with computer, such as the elderly. In addition, each test begins with a practice module, to assure understanding of the instructions. The domains and tests for each domain are (example stimuli for the tests are provided in Figure 2):

Figure 2.

Examples of test stimuli and procedures.

Abstraction and mental flexibility (ABF)

Penn Conditional Exclusion Test (PCET; Kurtz et al., 2004) is a measure of abstraction and concept formation, with alternate forms. Subjects decide which of 4 objects does not belong with the other 3 based on one of three sorting principles (e.g., shape, size, line thickness). Sorting principles change after 10 successive correct responses, and feedback is used to guide discovery of the principle and indicate its change. There are 4 alternate forms available. An accuracy score is calculated by multiplying the proportion of correct responses by the number of categories attained (out of 3 possible).

The PCET measures the ability to discover principles by hypothesis testing, where the principle shifts after its discovery is established. Such sorting tasks have been examined psychometrically (e.g., Miyake et al., 2000) and demonstrated robust involvement of frontal regions in extensive functional neuroimaging studies (e.g., Marenco et al., 1992; Adler et al., 2001; Crone et al., 2008; Specht et al., 2009). Individual differences in performance on such tasks have been related to frontal lobe volumes (Gunning-Dixon and Raz, 2003), dopaminergic influx (Vernaleken et al., 2007), and proneness to psychosis (Tallent and Gooding, 1999).

Attention (ATT)

The Penn Continuous Performance Test (PCPT; Kurtz et al., 2001) uses a standard CPT paradigm. The participant responds to a set of 7-segment displays presented 1/sec., whenever they form a digit (NUMBERS, initial 3 min) or letter (LETTERS, next 3 min). The number of true positive responses is recorded as the accuracy score and the median response time for true positive responses is the measure of attention speed.

The CPT is a well-established paradigm for measuring individual differences in vigilance and its developmental changes (e.g., Lin et al., 1999; Riccio et al., 2002). It has been related to activation of a frontal-parietal network (e.g., Adler et l., 2001; Ogg et al., 2008). The CPT task can help in the diagnosis of attention deficit disorders (Kooistra et al., 2009) and in detecting genetic susceptibility to psychosis (Chen and Faraone, 2009).

Working memory (WM)

The Letter N-Back (LNB; Ragland et al., 2002) presents letters for 500ms, and the participant has an additional 2000ms to respond by pressing the spacebar. There are three conditions: 0-Back - press the spacebar when the letter presented is an "X"; 1-Back - press when the letter presented is the same as the previous letter; 2-Back – press when the letter presented is the same as the one just before the previous letter. Following a training period, the test presents three blocks of each condition in a pre-determined order, for a total of 135 trials. The number of correct responses is recorded as the measure of accuracy and median response times for correct responses as a measure of speed.

The domain of “working memory” was conceptualized by Baddeley (2000) as an episodic buffer, where multimodal information is continuously integrated. It has been studied extensively with functional neuroimaging, and demonstrated to recruit dorsolateral prefrontal regions (Braver et al., 1997; Ragland et al., 1997; 2002; Rodriguez-Jimenez et al., 2009; Minzenberg et al., 2009). Working memory has been shown to have a significant genetic loading (Blokland et al., 2008), and to decline with aging in relation to the integrity of the dorsolateral prefrontal cortex and related regions (Mattay et al., 2006). Abnormal activation has been linked to schizophrenia (Callicott et al., 2003; Perlstein et al., 2003), bipolar illness (Drapier et al., 2008) and traumatic brain injury (Newsome et al., 2007),

Episodic Memory (MEM)

Verbal Memory. The Penn Word Memory Test (PWMT; Gur et al., 1997) presents 20 target words that are then mixed with 20 distractors equated for frequency, length, concreteness and low imageability. The participant’s score reflects the number of correctly recognized targets and correctly rejected foils. Median response times for correct responses serves as a measure of speed. A 20 min delayed recall procedure is also administered. Face Memory. The Penn Face Memory Test (PFMT; Gur et al., 1997) presents 20 digitized faces that are then mixed with 20 distractors equated for age, gender and ethnicity. The participant’s score reflects the number of correctly recognized targets and correctly rejected foils, and median response times for correct responses serves as a measure of speed. The procedure is repeated at 20 min delay. Spatial Memory. The Visual Object Learning Test (VOLT; Glahn et al., 1997) uses Euclidean shapes as stimuli with the same paradigm as the word and face. The participant’s score reflects the number of correctly recognized targets and correctly rejected foils, and again median response times for correct responses serves as a measure of speed. The procedure is repeated at 20 min delay. Two forms are available for each test.

This triad of declarative memory tasks examines the main visual modalities in an immediate and delayed recognition format. Such recognition memory tasks activate frontal and bilateral anterior medial temporal lobe regions (Gur et al., 1997; Jackson and Schacter, 2004). They have been applied extensively in functional neuroimaging studies both in healthy people and in patients with brain dysfunction (Gur et al., 1997). Performance on these tasks is heritable and deficits indicate vulnerability to neuropsychiatric disorders, including schizophrenia (Aliyu et al., 2006, Almasy et al., 2008; Greenwood et al., 2007) and brain disorders such as epilepsy (Saykin et al., 1989).

Language Reasoning (LAN; Gur et al., 1982)

The abbreviated Penn Verbal Reasoning Test consists of 8 verbal analogy problems from the Educational Testing Service (ETS) factor-referenced test kit. The number of correct responses is entered as the accuracy score. Two forms are available.

Verbal reasoning is a reliable measure of general intellectual abilities that has been shown to activate left temporo-parietal regions (Gur et al., 1982; 2000). Individual differences in performance have substantial heritabilities (Almasy et al., 2008; Greenwood et al., 2007). Reduced activation has been associated with aging (Gur at al., 1987a) and abnormal activation has been linked to brain dysfunction (Gur et al., 1987b).

Spatial Processing (SPA)

The Judgment of Line Orientation (JOLO) is a computerized version of Benton’s (Benton, Varney, and Hamsher, 1978) test (Gur et al., 1982). It presents two lines at an angle, and participants indicate the corresponding lines on a simultaneously presented array. The number of correct responses is entered as the accuracy score.

The JOLO is a reliable measure of spatial orientation abilities that has been shown to activate right temporo-parietal regions (Gur et al., 1982; 2000). Individual differences in performance have substantial heritabilities (Almasy et al., 2008; Greenwood et al., 2007). Reduced activation has been associated with aging (Gur at al., 1987a) and abnormal activation has been linked to brain dysfunction (Gur et al., 1987b).

Sensorimotor Processing Speed (SM)

The Mouse Practice task (MP; Gur et al., 2001a) requires moving the mouse and clicking as quickly as possible on a green square that disappears after the click. The square gets increasingly small. Since it is rare for participants to miss a target, no accuracy score is calculated and the median response time is used as the main dependent measure. Two forms are available. An un-speeded version of this task is administered at the beginning of the session to assure the participant has sufficient skills in moving and clicking the mouse.

Motor Speed (MOT)

The Computerized Finger Tapping Test (CTAP; Coleman et al., 1997; Almasy et al., 2008) measures how quickly the participant can press the spacebar using only the index finger. After a practice trial with each hand, the test presents five trials for the dominant hand alternating with five trials for the non-dominant hand. In each, the participant is asked to tap the spacebar repeatedly for 10s when the green "GO" screen is presented. The computer records the number of taps.

The finger-tapping procedure provides a measure of motor speed, which has been used in multiple functional neuroimaging studies because it maps well into motor and premotor regions contralateral to the moving finger, a response which diminishes with aging (Aizenstein et al., 2004). It correlates with other measures of motor speed, and shows a robust sex difference favoring males (Saykin et al., 1995).

Emotion Identification (EMO)

We used the 40-item facial affect identification Emotion Recognition test (ER-40; the method for its development was described earlier in this journal, Gur et al., 2002a). Facial displays of 4 emotions (Happy, Sad, Anger, Fear) and Neutral faces, 8 each, are presented and the subject identifies the emotion in a multiple-choice format. The facial stimuli are balanced for gender, age, and ethnicity. Number of correct identifications is used as an accuracy score, and median response times for correct responses serves as a measure of speed. Two forms are available.

Affect identification is the only test in this battery that measures social cognition, a domain of increased interest in behavioral neuroscience (Davidson et al., 2000). It has been applied in large normative populations (Mathersul et al, 2008; Williams et al., 2008), showing sex differences and age effects. Measures show high heritability, and deficit in facial affect identification have been linked to genetic susceptibility to neuropsychiatric disorders (Aliyu et al., 2006, Almasy et al., 2008; Greenwood et al., 2007). The facial affect recognition task has been applied in multiple functional neuroimaging studies and demonstrated to activates temporo-limbic regions (Gur et al., 2002b; Moser et al., 2007; Derntl et al., 2008), which diminishes with aging (Gunning-Dixon et al., 2003). Abnormal activation has been linked to schizophrenia (R.E. Gur et al., 2002; and depression Beesdo et al., 2009).

3. Results

3.1. Performance and internal consistency

The performance data for each test and information on its internal consistency are provided in Table 1. As can be seen, administration time of the CNB is approximately one hour and it yields moderate to high coefficient alpha values. Internal consistency estimates were higher for speed than for accuracy measures.

The intercorrelations among the tests are presented in Table 2. As can be seen, the measures are more highly intercorrelated for speed (lower triangle) than for accuracy (upper triangle). They also suggest greater intercorrelations within memory measures, while other correlations range from moderate to nil. These correlations thus support the construct validity of the battery.

Table 2.

Intercorrelations among the performance indices for tests included in the battery. The upper diagonal shows correlations with accuracy (hence none available for SM and MOT), and the lower diagonal shows correlations for RT. Correlations significant at the .01 level are italicized while those at p < .001 level are bolded. Test names as in Table 1.

ACCURACY\ SPEED	PCET	CPT N	CPT L	LNB	WORD IMM	WORD DEL	FACE IMM	FACE DEL	SPATIAL IMM	SPATIAL DEL	PVRT	CJOLO	SM	TAP	ER40
PCET	.	0.16	0.22	0.38	0.343	0.285	0.282	0.233	0.399	0.341	0.34	0.393	.	.	0.27
CPT N	0.31	.	0.32	−0.08	0.1	0.135	−0.01	0.069	0.106	0.209	0.04	0.12	.	.	0.16
CPT L	0.27	0.9	.	0.21	0.179	0.124	0.148	0.163	0.18	0.274	0.12	0.188	.	.	0.26
LNB	0.19	0.53	0.58	.	0.31	0.256	0.278	0.305	0.438	0.367	0.51	0.559	.	.	0.29
WORD IMM	0.42	0.31	0.31	0.31	.	0.389	0.318	0.227	0.307	0.273	0.34	0.265	.	.	0.2
WORD DEL	0.35	0.27	0.24	0.3	0.82	.	0.316	0.25	0.306	0.274	0.36	0.239	.	.	0.19
FACE IMM	0.39	0.28	0.28	0.21	0.604	0.582	.	0.608	0.338	0.337	0.46	0.286	.	.	0.31
FACE DEL	0.44	0.24	0.27	0.15	0.572	0.563	0.796	.	0.287	0.262	0.35	0.229	.	.	0.29
SPATIAL IMM	0.44	0.18	0.16	0.18	0.47	0.468	0.515	0.561	.	0.587	0.34	0.382	.	.	0.34
SPATIAL DEL	0.39	0.2	0.23	0.18	0.454	0.494	0.477	0.546	0.712	.	0.42	0.403	.	.	0.31
PVRT	0.23	0.28	0.23	0.28	0.248	0.244	0.412	0.32	0.252	0.321	.	0.632	.	.	0.36
CJOLO	0.56	0.42	0.4	0.28	0.418	0.351	0.439	0.399	0.389	0.418	0.4	.	.	.	0.35
SM	0.61	0.37	0.33	0.22	0.399	0.331	0.253	0.298	0.287	0.302	0.04	0.509	.	.	0.06
TAP	0.23	0.45	0.42	0.26	0.304	0.236	0.085	0.057	0.155	0.144	0.05	0.221	0.27	.	.
ER40	0.6	0.47	0.48	0.28	0.532	0.48	0.503	0.526	0.499	0.496	0.23	0.645	0.59	0.17	.

3.2. Sensitivity to dimensions of individual differences

Initial criterion validation of the battery included an examination of its sensitivity to dimensions of individual differences known to affect performance. We evaluated sex differences, correlations with age, and correlation with the socioeconomic indices of education and parental education.

Sex Differences

Men and women did not differ in age (mean±SD men 36.7±13.5, range 18–84; women 36.4±15.3, range 18–84, t = 0.25, df = 533, p = 0.8035), education (men 15.2±2.4, range 10 to 20; women 15.4±2.4, range 8 to 20, t = 0.87, df = 533, p = 0.3822) or parental education (men 13.8±3.4, range 3 to 20; women 14.1±3.1, range 2 to 20, t = 1.23, df = 533, p = 0.2188). Figure 3 shows means of males and females on the neurocognitive measures. As can be seen, men were more accurate and faster on the attention domain and the spatial task, and faster on the motor speed task, while women were more accurate in the working memory, word and face memory, and emotion processing tests.

Figure 3.

Mean (±SEM) of men (blue bars) and women (red bars) on the tests included in the battery. Domain names as in Table 1. Note that no accuracy measures are available for the sensorimotor test because no errors were made and for the motor speed test because no errors were possible. * p < .01; ** p < .001, red stars indicate women had better performance while blue stars indicate better performance in men. Note that accuracy and speed provide complementary information, and while performance is overall similar for men and women, some well known sex differences are replicated (better performance for women in memory and affect recognition and for men in spatial and motor domains), and some novel ones revealed (better performance for men on CPT and working memory).

Age Effects

Figure 4 shows correlations with age for both accuracy and speed, with error bars indicating 95% confidence intervals based on 1000 bootstrapped correlations. As can be seen, accuracy is negatively correlated with age for several measures, while response time increases with age for nearly all measures. The exception is working memory, which does not show reduced performance with aging either for accuracy or for speed. Immediate word memory, verbal reasoning and spatial processing do not show reduced accuracy with age.

Figure 4.

Correlations of age with accuracy (black bars) and response time (RT; gray bars) indices of performance on the tests. Error bars indicate 95% confidence intervals based on 1000 bootstraps. As seen, the effects of age are stronger for speed than for accuracy, and more pronounced for abstraction/flexibility and episodic memory than for attention and working memory.

Correlations with education and parental education

Figure 5 shows correlations with education (left panel) and parental education (right panel) for both accuracy and speed. As can be seen, education is associated with greater accuracy and speed for abstraction and flexibility, working memory and spatial processing, better accuracy for face and spatial memory, language reasoning and emotion processing, and faster motor speed. Correlations with parental education are generally higher and are seen both for accuracy and speed. Thus, parental education is associated with both better accuracy and speed for executive functions (abstraction, attention and working memory), immediate word memory, spatial and emotion processing, better accuracy in face and spatial memory, and faster sensorimotor processing and motor speed.

Figure 5.

Correlations of education (left panel) and parental education (right panel) with accuracy (black bars) and response time (RT; gray bars) indices of performance on the tests. Error bars were indicating 95% confidence intervals based on 1000 bootstraps. As seen, both education and parental education are associated with better performance, and the effects are stronger for accuracy than for speed. It also appears that the correlations are higher for parental education than for education, affecting both accuracy and speed.

4. Discussion

The results indicate the overall feasibility of using this brief computerized battery in research. A large sample of healthy volunteers was studied without difficulties; the average administration time was about an hour and the measures yielded moderate to high indices of reliability, construct validity and, more preliminarily, criterion validity.

Regarding reliability, the coefficient alpha values indicated acceptable to high internal consistency for all tests. They were expectedly higher for response times, for which they ranged from .777 for delayed spatial recognition to .969 for CPT numbers, than for accuracy, for which they ranged from .587 for the ER40 to .954 for total CPT. Thus, each test samples a coherent behavioral domain. While test-retest reliability is important to establish as well, and this effort is underway, high internal consistency indices auger positively for expecting good repeatability of measures. The high coefficient alphas suggest good reliability because, as generalization of the split-half reliability, they estimate the likelihood of obtaining the same scores under multiple simultaneous administrations (Allen and Yen, 1979).

The inter-correlations among tests were again generally higher for response time, where they ranged from .042 between SM and PVRT to .899 between CPT for numbers and CPT for letters, than for accuracy, where they ranged from −.075 between CPT for numbers and LNB to .632 between CJOLO and PVRT. The intercorrelations overall were moderate enough to indicate that the tests are diverse and do not measure the same domain repetitively. Notably, the highest correlation was observed between a spatial test, requiring the judgment of line orientation, and a purely language test, requiring verbal reasoning. They were both expected to measure higher order reasoning related to lateralized cortical processing and their correlation support the underlying model of brain-behavior relationships, thus supporting the construct validity of the battery.

The results of initial criterion validation were encouraging. The tests showed sensitivity to sex differences, with the expected better performance for males on spatial and motor tasks and better performance for females on memory and emotion processing tasks (Halpern et al., 2007). Less expected was the better performance for men on the CPT letters condition and for females on the LNB test. These are new findings of sex differences that suggest better vigilance abilities in men and better working memory abilities in women. These findings need replication.

The sensitivity of the tests to effects of normal aging provides further validation. Performance showed moderate to high correlations with age, and these were more pronounced for speed than for accuracy. Furthermore, as expected from the literature the effects of aging were more pronounced for memory and “fluid intelligence” tasks (abstraction and mental flexibility, attention) than for “crystallized intelligence” tasks (working memory, verbal reasoning, spatial processing). While older adults may have less experience with computers, the tasks were designed to require minimal computer skills from the participants. Furthermore, a training module familiarizes all participants with the computer interface and practice is provided before each task. Anecdotally, older adults were fully engaged and age effects were similar to paper and pencil measures.

Correlations with education and parental education were apparent, although they were generally higher for accuracy than for speed. More education was associated with better working memory, verbal reasoning and spatial processing skills, as well as better memory accuracy and faster motor speed. Higher parental education was associated with better performance across domains.

The overall results are positive, and the data presented here (and available to investigators, along with all the tests and a manual, at https://penncnp.med.upenn.edu) can be used as a normative database for comparison with a range of clinical populations. The data can also facilitate identification of young people at risk for brain disorders and may offer avenues for early intervention and rehabilitation. The computerized format offers many advantages over traditional measures. It provides an efficient means to obtain both accuracy and speed, requires minimal training of administrators, offers automated scoring and its procedures and measures are to experimental data obtained in functional neuroimaging. The CNB has been successfully implemented in large-scale genetic studies that, in further support of its validity, have reported significant heritability of performance across domains (Greenwood et al., 2007; Gur et al., 2007). Instructions have been translated into several languages and used in research including genetic studies and pharmacological trials. The incorporation of neurocognitive and affective neuroscience probes in genetically informative samples can provide the next stage of identifying brain systems that are targets for intervention, aiming to modify deficits observed in cognitive and emotion processing. Our current methodological efforts are to find the smallest number of items needed for each test, so as to eventually create even more efficient multiple forms of the battery for repeated studies.