Cognitive Abilities That Predict Success in a Computer-Based Training Program

Abstract

Purpose

The purposes of this study were (a) to identify cognitive abilities and other factors related to successful completion of training for computer-based tasks that simulated real jobs and (b) to create a brief assessment battery useful in assessing older adults for these kinds of jobs.

Design and Methods

Participants from three age groups (young, middle-aged, and older) completed a battery of cognitive measures. They then trained on one of three computer-based tasks that simulated actual jobs and were asked to perform the tasks for 3 days. We recorded whether they completed training and whether and how well they did the tasks. In a series of logistic regressions, we evaluated the ability of a subset of cognitive measures drawn from a larger battery to predict participants’ ability to successfully complete training and go on to task performance.

Results

Results confirmed theory-based expectations that measures of domain knowledge, crystallized intelligence, memory, and psychomotor speed would predict success in computer-based activities. A brief battery was able to predict older adults’ successful completion of training for one task but was less useful for another.

Implications

A brief battery of cognitive measures may be useful in evaluating individuals for job selection. Different measures are related to job-related criteria depending on task and group evaluated, although it was not possible to identify a reduced battery for one task. The specific cognitive abilities related to participants’ success have implications for task and interface design for the elderly population.

The population of the United States continues to age, and as a result of demographic trends fewer younger people are currently entering the job market. Older adults are thus likely to compose an increasingly important segment of the labor force during the coming decades. Although research has documented changes in older adults’ cognitive abilities with increasing age (Park & Schwarz, 2000), the possible relation of these changes to job performance has not been completely explored. A clearer understanding of how cognitive abilities are related to continued successful job performance among older adults may help maintain elders in some jobs while opening up training opportunities for others.

Computer-related tasks are common throughout most domains of work, and older adults are thus often required to use computers in their jobs. In 2001, nearly 34% of the older-than-65 workforce used a computer at work, whereas 21% of the same-age group used the Internet at work (Bureau of Labor Statistics, 2005). For workers in the 55- to 60-year-old group, demands for frequent work with computers doubled from 19% in 1992 to 40% in 2002 (Johnson, 2004). Previously published reports have confirmed that cognitive ability measures can be reliably and meaningfully related to performance on technologically based tasks relevant to elders’ work in technology-based situations (e.g., performing a data entry task or a simulated air traffic control scenario; Czaja & Sharit, 1998a; Czaja, Sharit, Ownby, Roth, & Nair, 2001; Morrow, 1994; Taylor, O’Hara, Mumenthaler, Rosen, & Yesavage, 2005).

The relation of older adults’ cognitive abilities to their ability to interact with computers may be relevant to their interactions with other similar devices in other arenas. Daily life increasingly requires all of us to deal with technology in such devices as microwave ovens, telephones with sometimes complex menu systems, or automated teller machines at banks (Rogers & Fisk, 2000). The ability to perform other technology-based activities (e.g., performing an Internet search for health care information) may enhance elders’ functioning and in the future may be important for health maintenance and for successful interactions with health insurance organizations (Rogers & Fisk, 2000). Previous reports have confirmed that cognitive ability measures can be reliably related to performance on technologically based tasks that can be important to elders’ quality of life (Fisk, Rogers, Charness, Czaja, & Sharit, 2004; Rogers & Fisk, 2000). A better understanding of how changes in older adults’ cognitive abilities affect their interactions with all these technological devices may improve experts’ ability to design more usable and effective interfaces.

Several studies have explored the potentially complex ways that older adults’ cognitive abilities may affect their interactions with computers and similar devices. Czaja and Sharit (1998a) used principal components analysis to extract three broad cognitive abilities from a large battery of measures. They then related individuals’ scores on these abilities (i.e., Visuomotor and Memory, Executive Functioning and Attention, and Processing Speed) to performance on a computer-based data entry task (recording mileage results from delivery truck runs). They also used a measure of each person’s experience with computers and age in predicting performance on this task. Analyses showed that computer experience, job knowledge, and the Visuomotor and Memory and Processing Speed scores were related to participants’ productivity (number of data records entered) and errors (number of mistakes made in the entries).

In another study (Czaja et al., 2001), researchers used structural equation modeling to create ability composites that were then related to task performance on an information search and retrieval task. This task simulated a job completed by customer service representatives of an insurance company, in which questions from customers were answered by telephone or e-mail. The job required that participants consult a database of information about various policies’ coverage, respond to questions, and document the questions and their resolution. The researchers evaluated participants’ success in this task as the number of queries successfully investigated and completed. They used structural equation modeling to create three cognitive ability composites: speed of processing, memory, and verbal speed and fluency. They used these composites, together with a measure of participants’ experience with computers and age, to predict task performance over several days. Each of the ability measures, as well as computer experience and age, was related to aspects of performance that included correct completion of task elements and navigational efficiency.

Comprehensive cognitive assessment batteries such those used in these studies are effective at assessing the relation of cognitive abilities to real-world computer task performance. Previous data analyses such as those reported by Czaja and Sharit (1998a) and Czaja and colleagues (2001) have shown that latent variables comprising multiple individual cognitive measures can be related to elders’ real-world performance on computer-related tasks. These studies have shown that broad dimensions of cognitive performance, in addition to computer experience and age, may be related to performance on real-world tasks that require interaction with computers or other technological interfaces. These findings are of considerable theoretical importance and provide the basis for additional exploration of how specific cognitive measures are related to performance.

Although these studies have shown a relation of abilities to level of performance, they have not explored the ability of cognitive ability measures to predict whether individuals are able to successfully complete training and perform a specific job—an important and substantially different question. In previous analyses, this issue was not addressed, because individuals who were not able to do the jobs by the nature of the study design could not contribute data to analyses of job performance. In addition, current assessment batteries are lengthy and typically require several hours for administration. A more practical battery of measures that would be potentially useful in research and in evaluation of older adults’ ability to perform jobs would be substantially shorter while having acceptable utility in predicting the likelihood of individuals’ success in specific jobs.

It may thus be useful to determine (a) what cognitive abilities are related to older adults successfully completing training for computer-related tasks and (b) whether a reduced set of measures can predict the successful completion of training for computer-based tasks.

Method

Data used for analyses in this study included cognitive assessment, demographic, and performance data on 417 individuals. More extensive data on this sample and the procedures used in this study are reported elsewhere (Czaja & Sharit, 1998b; Czaja et al., 2001). A total of 119 participants completed a data entry task (Task 1) that required only that data be transferred from handwritten sheets to a computer, 156 completed an information search task (Task 2) that required that they find and respond to information in a computer database, and 142 completed an accounts balancing task (Task 3) that required computation and error finding (see more extensive description of tasks in the next section). The average age of all participants was 50 years, but we recruited participants so that their ages fell into one of three groups (young, middle-aged, and older). Mean ages were 29.6 years for the young group, 49.6 years for the middle-aged group, and 68.2 years for the older group. The complete sample included 243 women and 174 men.

Experimental Tasks

With the collaboration of several U.S. corporations, we developed three computer-based tasks for use in this study. These included (a) data entry, (b) database inquiry, and (c) accounts balancing. We created the simulations so as to closely parallel the real-world tasks and thus to have ecological validity. The data entry task was based on a similar task performed at a large transportation company and involved entering trip record information into preformatted fields on a computer screen. Participants took information from written records provided by truck drivers and included such information as odometer readings, dates, states traversed, and fuel purchases. The main emphasis of this task was on speed and accuracy of data input with minimal demands placed on participants’ other cognitive skills.

The database inquiry task simulated a job performed by health insurance representatives. This task required participants to understand concepts about insurance plans and about computer-based access to health care plan information. Participants responded to simulated queries from plan members by telephone and on paper. They navigated through computer files that provided various categories of information and completed specific actions such as documenting and responding to members’ requests.

The accounts balancing task was a simulation of tasks performed by individuals who determine that customer accounts are in balance. Participants viewed screens from automatic check processing software to evaluate accounts that had been flagged as being out of balance, and they were required to determine the reason for the condition. After making a determination (e.g., a check was incorrectly scanned or a deposit slip was made out incorrectly), participants made the necessary corrections to ensure that accounts were correct.

Procedure

For each task, participants worked approximately 5 hr per day for 5 days. On Day 1, participants were screened and completed a baseline measure assessing experience with computers. We also administered ability measures in a prespecified order on Day 1. Participants received two 15-min rest breaks during battery administration and could take additional breaks if desired. On Day 2, participants were introduced to the computers used for simulations and trained on one of the tasks. They trained on tasks so that they could independently complete a set of practice problems and answer specific questions about the task. The focus of this assessment was to ensure that the participants successfully demonstrated the ability to perform the tasks and that they had acquired task-related knowledge at a level sufficient to inform their task performance. If they were unable to respond correctly to test questions on the first attempt, concepts missed were explained and the test was repeated. We allowed participants three attempts to pass the concept mastery test, after which we discontinued their participation if they were still unable to demonstrate mastery of task-related concepts at a level judged essential for task performance.

Participants who could not meet these training criteria after several attempts received $50 and their participation was terminated. Individuals completing the accounts balancing task received training on the use of a computer mouse and the Windows operating system. On Days 3 to 5, participants performed the task for which they had been trained for 3 hr each day.

Cognitive Battery

We chose the cognitive battery, administered to all participants, to tap a wide variety of potentially relevant abilities. These included cognitive processing speed; visuomotor skills; language and verbal fluency; abstraction; attention; and working, immediate, and long-term memory. We chose these measures to represent a broad range of abilities relevant to the tasks used in the study and the use of computer technology through an informal analysis of task requirements. For example, the data entry task emphasized speed of processing; the database inquiry task required understanding concepts related to health insurance, searching through data files, and integrating information from various databases. The accounts balancing task involved problem solving and mastery of mouse and Window operations. A number of cognitive abilities such as visuospatial skills and working memory are important to the successful performance of mouse and Window tasks (Czaja & Lee, 2003). Table 1 lists measures included in the battery with descriptions.

Table 1.

Cognitive Ability Measures

Measure	Ability	Description
WAIS-R Vocabulary (Wechsler, 1981)	Vocabulary	Participants orally define 35 words.
Controlled Oral Word Association (Benton & Hamsher, 1989)	Verbal fluency	Participants say as many words as they can that begin with the letters C, F, or L.
Trail Making Test, Forms A and B (Reitan, 1958)	Form A, visuomotor speed; Form B, executive function and psychomotor speed	In Part A, participants draw lines to connect numbered circles; in Part B, participants alternate the order of numbers and letters while connecting the circles.
WAIS-R Digit Span (Wechsler, 1981)	Attention	In the first part, participants repeat increasingly long sequences of numbers orally presented to them; in the second part, participants are presented with series of digits and must repeat them backwards.
Wechsler Memory Scale, Visual Reproduction, Immediate and Delayed (Wechsler, 1987)	Immediate and long-term visual memory	Participants are presented geometric designs on cards and immediately afterward are asked to draw the design (immediate recall). After 30 min, they are asked to draw the same designs from memory (delayed recall).
California Verbal Learning Test, Trials 1–5 and Delayed Recall (Delis et al., 1987)	Immediate and long-term verbal memory	Participants are orally given a list of words and asked to remember them. The list is presented five times (immediate recall). After 20 min, participants are asked to recall the same list (delayed recall).
WAIS-R Digit Symbol (Wechsler, 1981)	Visuomotor speed	Participants are presented with a key that pairs numbers with meaningless symbols and are asked to fill in blanks under rows of numbers that do not have symbols.
Grooved Pegboard (Tiffin, 1968)	Manual dexterity and psychomotor speed	Participants place pegs into holes in a board as rapidly as they can.
Figural Visual Scanning and Discrimination (Ekstrom et al., 1976)	Processing speed	Participants review geometric forms printed on a piece of paper and match them against a standard.
Sternberg Short-Term Memory Search Tasks (Sternberg, 1975)	Processing speed	A group of digits is presented, then a probe digit. Participants are then asked whether the probe digit was in the first group. Reaction time is measured.
Two Choice Visual Reaction Time (Wilkie et al., 1990)	Processing speed	Participants respond to a stimulus that appears on a computer screen with their right or left hand, depending on where the target is presented on the screen.

Note: WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Data Analysis

The primary outcome measure was the ability of participants to complete training, meet criteria for task mastery, and go on to task performance. We chose this criterion to provide information that might be useful in selecting potential trainees, as we thought that employers might want to better understand what factors are related to successful completion of training and choosing to continue to begin a job. In order to explore the relations between abilities, age, computer experience, and education and outcome (successfully completing training and going on to task performance), we developed logistic regression models for the tasks in a two-stage process. In the first stage, a bootstrapping procedure identified the 10 measures that were most often associated with the outcome variables for the groups of younger and older participants. Next we created batteries with progressively fewer measures by deleting one measure at a time sequentially. We assessed the performances of these reduced batteries for their usefulness in selection balanced with battery length. We determined optimal battery size through evaluation of several criteria, as described later, and used the battery to evaluate younger and older adults’ performance. These analyses allowed for comparisons of younger and older participants. This grouping allowed us to address a central question for this study: What differences, if any, existed between the two younger groups and the older group? Our rationale for this grouping is simply that individuals in the two younger groups might routinely be considered for employment, whereas those involved in job selection might consider the individuals in the older group less suitable for computer-related jobs.

We used an empirical ranking strategy in order to evaluate which measures were most closely related to successfully completing training. The strategy bootstrapped stepwise selection methods in logistic regression analyses of all available variables to predict the outcome in younger and older adults. Consistent with the recommendations and findings of Steyerberg and others (Shtatland, Kleinman, & Cain, 2004; Steyerberg, Eijkemans, Harrell, & Habbema, 2000; Steyerberg et al., 2001), we addressed the problems associated with the use of stepwise methods, such as sensitivity of variable selection to small changes in sample and instability of results, through the use of a bootstrapping procedure. In this procedure, we entered all candidate variables from the battery into a routine (swboot) available for the STATA software package (StataCorp, College Station, TX). This routine completes a large number of model estimations via a stepwise selection procedure using random samples drawn from the existing data for each repetition. Consistent with others’ recommendations (Shtatland et al., 2004), probabilities used for variable entry and exit from the model were large so as to capture the greatest number of candidate variables. We ran the routine for 1,000 repetitions for the groups of younger and middle-aged participants combined and for the group of older participants. We used the rank order of how frequently measures were significant predictors in regression models as an indicator of the importance of measures in predicting the criterion and thus as an index of the importance of the participant characteristics and cognitive abilities in training success.

Candidate variables for the reduced battery were the 10 measures with highest mean ranks in these bootstrapped models. We chose 10 measures in order to ensure adequate coverage of abilities elucidated in an earlier factor analytic study (Czaja et al., 2001) while reducing the complexity of the subsequent analyses. We evaluated batteries with fewer measures by creating models first with all 10 variables and then in subsequent analyses deleting the variable with the smallest Wald chi-square value, rerunning the model, and reevaluating the revised model. In maximum likelihood estimation as we employed in this study, the Wald statistic provides a measures of the magnitude of the relation between independent and dependent variables. We used the Wald chi-square value here as an estimate of the magnitude of the relation between each of the tests in the reduced battery and the outcome they predict, controlling for the influence of the other measures. It can be considered a standardized measure of the effect size for each of the measures in the battery and allows for a comparison of these effects across measures in each subset. It thus provides a basis for elimination of each of the measures in sequentially reduced batteries.

The number of predictors in the model was thus reduced from 10 to 1. At each stage, we recorded indicators of model fit and predictive capacity and subsequently graphed them to allow for determination of the optimal combination of measures. For each model, we recorded Akaike’s information criterion (AIC; an index of model fit to the data adjusted for the number of parameters in the model), the area under the curve (AUC) for a receiver operating characteristics curve, and the R² value. We created plots of number variables by AIC and R² to assess the relative usefulness of models with varying numbers of predictors in relation to the effectiveness of the prediction (R²) and fit to the data (AIC). We anticipated that inspection of these plots would reveal asymptotes for each of these indexes that would allow for determination of the optimal number of measures to include in a reduced model.

Finally, we completed separate analyses with the reduced battery for Tasks 2 and 3 for the combined group of younger and middle-aged individuals and for the group of older participants. We did this so that the analyses would address a central question for this study: What differences, if any, were there between the two younger groups and the older group? We did not include Task 1 in this process because of the small number of individuals who failed to complete it, presumably because of its simplicity. These analyses allowed for a further investigation of the relations of specific cognitive abilities to the two outcomes in different task contexts.

Results

Table 2 reports demographic data for the sample. Tables 3, 4, and 5 present reasons for why participants did not complete their participation. The relations between reason for not completing a task and age group or gender were not statistically significant. Older participants were thus not more likely to choose not to complete training or task completion than were their younger counterparts. The relation between task assignment and successful task completion, however, was significant. Inspection of the frequencies in Table 4 suggests that this may be due to fewer individuals having been disqualified or choosing not to continue in the data entry task (Task 1) compared with the other two tasks.

Table 2.

Age and Gender Distribution of Age Groups by Task Assignment

	Task
Groupa	Data Entry	Database Inquiry	Accounts Balancing	Total for Age Group
Young	29.68 (5.69)	29.42 (6.24)	29.65 (5.75)	29.57 (5.88)
(68 men & 70 women)	40	52	46	138
Middle-aged	48.74 (5.18)	48.93 (6.65)	50.91 (5.63)	49.58 (5.89)
(42 men & 80 women)	38	49	44	122
Older	66.98 (8.69)	68.09 (4.01)	69.46 (4.52)	68.25 (5.79)
(64 men & 93 women)	41	64	52	157
Total	48.61 (16.84)	50.29 (17.55)	50.82 (17.38)	49.99 (17.27)
(174 men & 243 women)	119	156	142	417

Notes: Data are mean age in years (SD), with n underneath. Analysis of variance for age across task groups, F(2, 414 = 0.61, p = .55.

^aχ²(2, N = X417 = 5.97, p = .05; the gender distribution for the younger group was more nearly 1:1 than for the two older groups, so that proportionally more women participated in the two older groups.

Table 3.

Reasons for Not Completing Tasks by Age Group

	Finishing Status
Age Group	DQ Coga	DQ Trainb	DQ Taskc	Chose Not to Continue (Cognitive)d	Chose Not to Continue (Training)e	Chose Not to Continue (Task)f	Total Disqualified	Total Chose Not to Continue	Completedg	Total
Young	4	7	1	2	8	3	12	13	113	138
Middle-aged	2	11	1	2	11	1	14	14	94	122
Older	1	15	1	3	19	9	17	31	109	157
Total	7	33	3	7	38	13	43	58	316	417

Notes: χ²(12, N = 417 = 15.15, p = .23.

^aDQ Cog = disqualified (cognitive). Participant did not meet criteria for participation on cognitive measures (e.g., cutoff score on a memory measure).

^bDQ Train = disqualified (training). Participant did not meet criteria after multiple exposures to the training phase for each task.

^cDQ Task = disqualified (task). Participant did not meet criteria for continuing in task phase.

^dParticipant elected not to continue after beginning the cognitive battery but before beginning the training phase.

^eParticipant elected not to continue after beginning the training phase but before beginning the task phase.

^fParticipant elected not to continue after beginning the task phase but before completing it.

^gSuccessfully completed all phases of training and task performance.

Table 4.

Reasons for Not Completing Tasks by Task Assignment

	Finishing Status
Task	DQ Coga	DQ Trainb	DQ Taskc	Chose Not to Continue (Cognitive)d	Chose Not to Continue (Training)e	Chose Not to Continue (Task)f	Total Disqualified	Total Chose Not to Continue	Completedg	Total
Data entry	2	3	1	3	3	2	6	8	105	119
Database inquiry	3	17	0	4	21	0	20	25	111	156
Accounts balancing	2	13	2	0	14	11	17	25	100	142
Total	7	33	3	7	38	13	43	58	316	417

Notes: χ²(12, N = 40.05, p < .001.

^aDQ Cog = disqualified (cognitive). Participant did not meet criteria for participation on cognitive measures (e.g., cutoff score on a memory measure).

^bDQ Train = disqualified (training). Participant did not meet criteria after multiple exposures to the training phase for each task.

^cDQ Task = disqualified (task). Participant did not meet criteria for continuing in task phase.

^dParticipant elected not to continue after beginning the cognitive battery but before beginning the training phase.

^eParticipant elected not to continue after beginning the training phase but before beginning the task phase.

^fParticipant elected not to continue after beginning the task phase but before completing it.

^gSuccessfully completed all phases of training and task performance.

Table 5.

Reasons for Not Completing Tasks by Gender

	Finishing Status
Group	DQ Coga	DQ Trainb	DQ Taskc	Chose Not to Continue (Cognitive)d	Chose Not to Continue (Training)e	Chose Not to Continue (Task)f	Total Disqualified	Total Chose Not to Continue	Completedg	Total
Men	3	15	3	2	14	5	21	21	132	174
Women	4	18	0	5	24	8	22	37	184	243
Total	7	33	3	7	38	13	43	58	316	417

Notes: χ²(6, N = XX) = 5.31, p = .51.

^aDQ Cog = disqualified (cognitive). Participant did not meet criteria for participation on cognitive measures (e.g., cutoff score on a memory measure).

^bDQ Train = disqualified (training). Participant did not meet criteria after multiple exposures to the training phase for each task.

^cDQ Task = disqualified (task). Participant did not meet criteria for continuing in task phase.

^dParticipant elected not to continue after beginning the cognitive battery but before beginning the training phase.

^eParticipant elected not to continue after beginning the training phase but before beginning the task phase.

^fParticipant elected not to continue after beginning the task phase but before completing it.

^gSuccessfully completed all phases of training and task performance.

Only a small number of participants failed to complete training for Task 1 (data entry), and the logistic regression model for this task was associated with a low R² value. This suggested that very little of the variability in completing training could be predicted for this task, presumably because of the small number of participants who did not complete this task (n =14) and the small number of people who did not achieve training criteria (see Table 4). Because of these findings, we report no additional analyses for this task.

The bootstrapping procedure provided data on the number of times each candidate variable from the entire battery was included in prediction models over 1,000 bootstrapped replications. The model thus used age, education, and results of the computer experience questionnaire together with cognitive measures to predict participants’ having successfully completed training and having gone on to task performance. Table 6 presents results of the bootstrapping procedure for the young and middle-aged sample predicting success on both tasks. Analyses of smaller batteries of measures then used the highest ranking 10 measures. Of measures used in the battery, age, education, domain knowledge (Computer Experience Questionnaire), long-term visual memory (Wechsler Adult Intelligence Scale–Revised[WAIS-R] Visual Reproduction, Delayed Recall), psychomotor speed (Trail Making Test, Part B time; WAIS-R Digit Symbol; Grooved Pegboard; and Figural Visual Scanning and Discrimination), short-term visual memory (WAIS-R Visual Reproduction, Immediate Recall), and crystallized knowledge (WAIS-R Vocabulary) were the predictors most often included in predictive models.

Table 6.

Frequency of Measures’ Appearance in Bootstrapped Models for Younger and Middle-Aged Participants for Tasks 2 and 3

Measure	Frequency
Age	812
Computer experience	698
Education	647
WAIS-R Vocabulary	865
WAIS-R Digit Symbol	744
WAIS-R Digit Span	570
Wechsler Memory Scale–Revised Visual Reproduction, Immediate Recall	671
Wechsler Memory Scale–Revised Visual Reproduction, Delayed Recall	971
Figural Visual Scanning and Discrimination, total time	642
Trail Making Test, Part B time	886
Grooved Pegboard, dominant hand time	617
Controlled Oral Word Association	561
California Verbal Learning Test, learning trial	519
California Verbal Learning Test, recall trial	605

Note: Items in bold were included in analyses of the battery to predict participants’ completion of training and going on the task phase of the study. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

We developed logistic regression models using these 10 measures for the outcome criterion, successfully completing training. We then evaluated these models by eliminating at each step the single measure that contributed the smallest amount to prediction as assessed by its Wald chi-square value. Inspection of plots of measures of model fit (AIC) and the amount of variance explained by the battery (R²) allowed for evaluation of combinations of these indexes for varying numbers of measures in the model. There was thus a point at which adding more measures contributed little to prediction (as shown by small increments in R² value) while adding complexity to the regression model (increasing values of the AIC). It was thus possible to determine the optimal number of measures to include in a reduced model, balancing model prediction and complexity. Figure 1 presents the plot of R² versus AIC for batteries with 10 to 1 predictors. In this instance, the battery predicted successful training on Task 2 for the combined group of younger and middle-aged individuals.

Figure 1.

Plot of Akaike’s information criterion (AIC) and R² values for different numbers of predictors.

For both Tasks 2 and 3, plots indicated that models that included from six to eight variables provided optimal combinations of efficiency and predictive power (see Figure 1). The plot shows the relation of model fit (R², on the left axis) to the AIC value (on the right axis) for models with different numbers of predictors. As might be expected, the AIC value initially improves substantially (smaller values indicating better fit) with the addition of variables to the model. It reaches an asymptotic low at six variables, then slowly increases with the addition of variables (the AIC includes a penalty for model complexity). The R² value, representing the amount of variability in the outcome variable predicted by the model, is plotted on the left axis of the graph. It initially increases sharply with the addition of variables to the model but also reaches an asymptote between six and eight variables, at which point the addition of variables to the model adds little to its value.

Tables 7 and 8 present regression analyses of the smaller number of measures prediction for Tasks 2 and 3, respectively, for the two younger groups. The optimal model for Task 2 was associated with an R² value of 0.39 and an AUC value of 0.82. The model for Task 3 was associated with an R² value of 0.24 and an AUC value of 0.81. The reduced battery was thus useful in predicting the ability of younger participants to successfully complete training.

Table 7.

Reduced Battery Predicting Successful Training for Younger Groups for Task 2

Measure	Coefficient	SEa	χ2	p
Education	0.24	0.17	1.97	.87
WAIS-R Vocabulary	0.02	0.02	0.89	.16
Wechsler Memory Scale, Visual Reproduction Delayed Recall	0.19	0.11	2.95	.09
Figural Visual Scanning and Discrimination	0.04	0.02	4.44	.04
Trail Making Test, Part B time	−0.03	0.01	5.57	.02
Grooved Pegboard dominant hand time	−0.03	0.02	2.33	.13

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 92 participants in the two younger groups who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.82, R² = 0.39. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Table 8.

Reduced Battery Predicting Successful Training for Younger Groups for Task 3

Measure	Coefficient	SEa	χ2	p
Education	0.04	0.19	0.41	.52
WAIS-R Vocabulary	0.07	0.03	5.00	.03
Wechsler Memory Scale, Visual Reproduction Immediate Recall	−0.36	0.21	2.91	.09
Wechsler Memory Scale, Visual Reproduction Delayed Recall	0.55	0.22	6.53	.01
Figural Visual Scanning and Discrimination	−.004	0.02	0.06	.80
Grooved Pegboard dominant hand time	−0.01	0.02	0.35	.56

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 90 participants in the two younger groups who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.81, R² = 0.24. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Table 9 reports the number of times specific measures significantly predicted task completion for both tasks for the older sample alone. Of measures used in the battery, age, education, domain knowledge (Computer Experience Questionnaire), long-term visual memory (WAIS-R Visual Reproduction, Delayed Recall), psychomotor speed (Trail Making Test, Part B time; Grooved Pegboard; and Figural Visual Scanning and Discrimination), short-term memory (WAIS-R Visual Reproduction, Immediate Recall; WAIS-R Digit Span), and crystallized knowledge (WAIS-R Vocabulary) were the measures most often included in predictive models.

Table 9.

Frequency of Measures’ Appearance in Bootstrapped Models for Older Participants for Tasks 2 and 3

Measure	Frequency
Age	814
Computer experience	647
Education	568
WAIS-R Vocabulary	865
WAIS-R Digit Symbol	570
WAIS-R Digit Span	744
Wechsler Memory Scale–Revised Visual Reproduction, Immediate Recall	671
Wechsler Memory Scale–Revised Visual Reproduction, Delayed Recall	971
Figural Visual Scanning and Discrimination, total time	642
Trail Making Test, Part B time	886
Grooved Pegboard, dominant hand time	617
Controlled Oral Word Association	561
California Verbal Learning Test, learning trial	519
California Verbal Learning Test, recall trial	605

Note: Items in bold were included in analyses of the battery to predict participants’ completion of training and going on the task phase of the study. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

We followed a procedure similar to that used with younger participants to evaluate regression models including 10 to 1 variables, in this case using the 10 variables that were most likely to predict successful completion of training among the older participants. Again, models that included from six to eight predictors were associated with optimal combinations of predictive ability and complexity. Tables 10 and 11 present these regression models. For Task 2, age, computer experience, education, short-term visual memory, and visual discrimination speed predicted successful completion of training. For Task 3, only one of the measures, the WAIS-R Digit Span subtest, approached significance, reflecting verbal short-term memory. The optimal model for Task 2 was associated with an R² value of 0.56 and an AUC value of 0.90. The model for Task 3 was associated with an R² value of 0.22 and an AUC value of 0.76.

Table 10.

Reduced Battery Predicting Successful Training for Older Group for Task 2

Measure	Coefficient	SEa	χ2	p
Age	0.38	0.15	6.20	.01
Computer Experience Questionnaire	0.24	0.11	4.68	.03
Education	0.50	0.24	4.31	.04
WAIS-R Digit Span	0.20	0.11	3.28	.07
Wechsler Memory Scale, Visual Reproduction Immediate Recall	0.64	0.23	7.49	.01
Figural Visual Scanning and Discrimination	−0.09	0.04	6.78	.01

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 64 participants in the older group who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.90, R² = 0.56. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Table 11.

Reduced Battery Predicting Successful Training for Older Group for Task 3

Measure	Coefficient	SEa	χ2	p
Age	0.06	0.09	0.42	.58
Computer Experience Questionnaire	0.09	0.08	1.34	.25
WAIS-R Vocabulary	0.03	0.05	0.47	.49
WAIS-R Digit Span	−0.16	0.08	3.54	.06
Wechsler Memory Scale, Visual Reproduction Immediate Recall	−0.02	0.02	1.04	.31
Trail Making Test, Part B time	−0.01	0.01	0.44	.51

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 64 participants in the older group who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.76, R² = 0.22. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Because of the role of computer experience in predicting training success for older but not younger participants, we evaluated between-group differences in this variable. There was a significant between-group difference, F(2, 389=16.27, p < .001. Post hoc tests with the Tukey procedure revealed no differences in computer experience for the two younger groups (p = .20) but significant differences between the two younger groups and the older group (ps < .001). Within the older group itself, the correlation of computer experience and age was significantly negative (−0.30, p = .04). Finding this difference led us to consider whether age might interact with other variables in the regression models. We included all Age × Variable interactions in additional analyses, and we included in Tables 12 and 13 those that affected models in substantive ways (either by being significant themselves or by affecting other predictor variables). By including this interaction, we reduced the effect of age by itself on successful completion while clarifying other ability–outcome relations. A reduced battery was thus more useful in predicting successful completion of training for older participants working in the database inquiry task and less useful for participants in the accounts balancing condition.

Table 12.

Reduced Battery Predicting Successful Training for Older Group for Task 2 With Age × Variable Interaction

Measure	Coefficient	SEa	χ2	p
Age	−0.77	0.58	1.73	.19
Computer Experience Questionnaire	0.03	3.16	0.00	.99
Education	0.73	0.33	4.74	.03
WAIS-R Digit Span	−5.59	2.85	4.00	.05
Wechsler Memory Scale, Visual Reproduction Immediate Recall	0.91	0.31	8.73	.003
Figural Visual Scanning and Discrimination	−0.12	0.04	7.57	.01
Age × Computer Experience	0.004	0.03	0.02	.90
Age × WAIS-R Vocabulary	0.09	0.04	4.27	.04

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 64 participants in the older group who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.92, R² = 0.65. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Table 13.

Reduced Battery Predicting Successful Training for Older Group for Task 3 With Age × Variable Interaction

Measure	Coefficient	SEa	χ2	p
Age	−0.03	0.12	0.04	.85
Computer Experience Questionnaire	−1.50	1.37	1.20	.27
WAIS-R Vocabulary	0.04	0.05	0.79	.38
WAIS-R Digit Span	−0.20	0.10	4.36	.04
Wechsler Memory Scale, Visual Reproduction Immediate Recall	0.11	0.14	0.55	.46
Trail Making Test, Part B time	−0.01	0.01	1.72	.19
Age × Computer Experience	0.02	0.02	1.34	.25

Note: Model selection based on progressive elimination of least important predictor based on the Wald chi-square value for the 64 participants in the older group who completed training on Task 2 and went on to the task performance phase. Model area under the curve = 0.77, R² = 0.24. WAIS-R = Wechsler Adult Intelligence Scale–Revised.

Discussion

The purpose of this study was to further explore the cognitive abilities related to the successful completion of computer-related training in younger and older individuals. A secondary purpose was to determine whether a less comprehensive and less time-consuming battery of cognitive measures than has been previously employed would be useful in evaluating individuals’ ability to complete training for these tasks. Age and measures of crystallized intelligence, visual memory, psychomotor speed, and domain knowledge predicted older participants’ successful completion of training. Additional analyses showed that a reduced battery was potentially useful in predicting who would succeed in training.

One of the goals of this study was to determine if the relationship between abilities and performance varied as a function of task demands. This type of information is important to the development of training programs and selection criteria. Results showed that although several measures were clearly related to older participants’ success in training for the database inquiry (Tables 10 and 12), the same measures were less clearly related to the accounts balancing task (Tables 11 and 13). This finding suggests that other factors not included in the analysis may have been important in determining training success for this task in the older participants. We also saw this pattern in analyses of the complete battery (data not shown here due to space limitations). When broad ability factors developed in structural equation models were used to predict level of performance on these tasks (Czaja et al., 2001), different patterns of predictors were obtained. Future research should thus focus on obtaining an improved understanding of the factors related to older adults’ success in the sort of complex task (involving computer skills, information search, and conceptual understanding) represented by the accounts balancing training. We note that the apparent relation between training success and age (Table 10) disappeared when we took into account the Age × Variable interactions (Table 12). It may not be age in itself that affects performance in computer-related tasks, but age-related changes in other variables such as computer experience and cognitive abilities.

These results complement those reported earlier (Czaja et al., 2001) that examined the relation of broad cognitive ability dimensions, estimated via structural equation modeling from the same battery, on the level of performance on these same three tasks. That study showed that age and computer experience and the cognitive domains of processing speed, memory, and verbal skills predicted level of performance in participants who completed all parts of the study. The brief batteries developed in this study include the same measure of computer experience as a predictor not only of level of performance but also of older adults successfully completing training for Tasks 2 and 3. This highlights the relation of domain knowledge to success in this task. The earlier study also showed that speed of processing was related to level of performance. Here again, the reduced battery developed here includes two measures for which speed is an essential component (Figural Visual Scanning and Discrimination and Trail Making Test, Part B). As before, memory is implicated in both successful completion and level of performance, and in our current study visual memory was important in successful training for Task 2.

The importance of computer experience in both successful completion of training and in participants’ level of performance emphasizes the importance of knowledge about computers and related skills for older adults to be successful in jobs that require computer use. Our finding that this variable was inversely related to age among older participant underlines the importance of this experience for older computer users. This finding has implications for training programs and suggests that it may be important to assess trainees’ baseline computer skills and provide for additional training when necessary. This may be critical prior to actually training job candidates on specific tasks. Failing to do so, we speculate, might decrease trainees’ willingness or ability to complete training.

The inclusion of two measures that reflect psychomotor speed in the reduced battery (Trail Making Test, Part B; and Figural Visual Scanning and Discrimination) implies that older adults may be at a comparative disadvantage to younger adults as a result of many older adults’ slower speed of processing (Salthouse, 1996). Jobs that place a premium on speed may thus be less suitable for older adults, or older adults’ relatively slower psychomotor speed should be taken into account when designing and training for this sort of job. In this connection as well, we note that simple memory aids have facilitated elders’ performance on memory-related tasks (Sharit, Czaja, Nair, & Lee, 2003). Simple job modifications that take elders’ unique cognitive abilities into account might enable older workers to perform some jobs.

These results thus demonstrate that specific cognitive measures may be useful in predicting individuals’ probability of completing training on and performing technology-related tasks. Developers of test batteries should note that the performance of different measures may vary by task—it may be necessary to validate test batteries for specific tasks. The finding of substantial differences between predictive batteries for the samples of younger and middle-aged and for older adults also has implications for battery development and implementation. It may be important to develop and validate test batteries for vocational assessment and placement specifically for older workers.

The battery thus may also show differential functioning depending on age group and on the characteristics of the task performance predicted. Measures that were significant predictors of successfully completing training for the younger groups tapped crystallized intelligence, memory, and psychomotor speed. Individual measures of these domains were useful in the reduced battery for predicting who would successfully complete training. In contrast, successful completion for older adults was only predicted by age, computer experience, and measures of short-term memory and psychomotor speed, and the significant relation of training success to age and computer experience was not present in models that included terms for the interaction of age with these variables. It is thus likely that it is not age in itself but age-related differences in variables such as computer experience that are important in understanding how to help older workers be successful in computer-related jobs. This is a distinction that makes a difference, as it suggests that additional training and modifications in task design may enable older workers to complete tasks that otherwise might be difficult or frustrating for them.

We should acknowledge several limitations of this study. Although our sample size was originally powered for detection of differences across the three age groups, it was not powered to detect differences within groups. Although the sample size we employed was probably sufficient to detect significant predictors, a sample size of 10 events (the predicted occurrence) is a widely used rule of thumb in evaluating sample size for logistic regression. A simulation study by Peduzzi and colleagues (1996) showed that sample sizes of 5 to 10 events per variable included in a regression equation produced fairly stable coefficients in logistic regression models. All of the models reported in this article met the criterion of 10 events per variable with the exception of the analyses for Tasks 2 and 3 within the older group, which met the criterion of 5 events per variable. Given this issue, readers should interpret cautiously the results of these analyses. This may also be in part a reason for the lack of finding several predictors for the model for older adults training for Task 3. It is possible that a larger sample it might have been able to detect predictors. It may also be possible, however, that other relevant predictors related to Task 3 were not included in our model and thus were not identified.

These results suggest that after taking domain knowledge and specific cognitive abilities into account, age in itself is not a predictor of completion success. This finding has important implications for training and task modification for older workers. It is not that older workers cannot succeed at computer-related tasks, but that age-related changes in cognitive abilities such as psychomotor speed should be taken into account when designing training and job-related tasks to be performed by older workers. Computer literacy is high in many individuals aged 50 years and older, and as these individuals grow older, it will be even more important to accommodate work situations for them. Results of these analyses thus provide insights into the potentially complex relations among our participants’ knowledge, cognitive abilities, and performance on computer-based tasks.