Expanding horizons in ergonomics research

Introduction

Ergonomics, the use of scientific thinking in the design of products and of working environments, has undergone three major waves of research innovations and applications. The first owed a great deal to Paul Fitts, who was a member of the generation of scientists coming to maturity during World War II. Below I have labeled it with the name Human Performance, which he gave to the basic science that could be applied to human factors in the work environment. The second was influenced by Herbert Simon and the development of cognitive psychology, leading to cognitive engineering in the design process. Although Fitts had passed away by the time cognitive engineering research was underway, it still made close contact with his approach. This special issue reflects a new development in ergonomics that Parasuraman (2011) has called Neuroergonomics, involving the application of brain science to human factors. Although each wave has added valuable new perspectives, as discussed briefly below, the contributions of Paul Fitts and of information theory still remain relevant (Gleick, 2011).

Human performance

During World War II Paul Fitts was a young researcher engaged in studies attempting to improve the effectiveness of the Air Force. During this intense period, Fitts had three ideas that shaped the development of Ergonomics and influenced subsequent theories of Psychology.

The first was that input information supplied to the pilot and the responses they produced to control the flight could be structured in such a way as to be compatible with each other. This idea led to Fitts’ famous studies on the role of stimulus response compatibility and population stereotypes on the speed of human information processing (Fitts and Deininger, 1954). Not only did high stimulus response compatibility reduce the time to respond, but it also greatly reduced errors and improved the time to learn new codes. Every designer of equipment to be used by humans should be aware of the myriad of studies attesting to the importance of these considerations in the design process.

A second important idea that Fitts had was that detailed protocols of the pilot actions in accidents and near misses could reveal design flaws (Fitts, 1951). These protocols had significant influence on the design of aircraft cockpits of that era. The development by Fitts of critical incident analysis was an important forerunner to the effort to develop detailed protocols of experts that has become an important part of the cognitive design process (Ericsson and Simon, 1980).

Fitts’ third idea was that information derived from these ergonomic studies could lead to a science of human information processing that he called human performance theory. This theory could specify human limitations and capacities in purely objective quantitative engineering terms. The metric most frequently used was information theory, which had then been recently developed by Shannon and Weaver (1949). Possibly the triumph of this approach was the ability to predict the time to initiate a response based on the information transmitted (Hick-Hyman law) and the movement time to the target based on the distance moved and accuracy of termination (Fitts Law). After Paul Fitts passed away I completed a small volume based on his conception of human performance theory (Fitts and Posner, 1967).

Cognitive engineering

The quantitative information theory approach found in the book Human Performance was created at a time when psychology was mainly conceived as a science of behavior. Although Fitts was far from a behaviorist, his stress upon input and output information was certainly designed to make the information processing approach appealing to those trained in the behavioral tradition. Shortly after Fitts passed away in 1965, a book by Ulrich Neisser called Cognitive Psychology (1967) marked the start of the new cognitive era in psychology. Neisser prematurely declared the end of the era of information processing and built the new cognitive psychology more upon computer simulations of complex processes from Herbert Simon and experimental methods designed to probe the nature of conscious and unconscious decisions. Cognitive Engineering was an outgrowth of the new cognitive psychology. Unlike human performance theory, cognitive engineering could be widely applied to situations in which the person develops a mental model of a complex situation. Most areas of the design of industrial and consumer products could be seen as appropriate to the Cognitive Engineering idea.

I prefer to think of Cognitive Engineering as an enlargement of the enterprise that Fitts began, not as a replacement. For example, John Anderson and his associates (2007) showed that the learning of many tasks could be seen as trying out and discarding strategies for the performance of the task. This view led to the prediction of a power function relating training time to performance and the results of older studies summarized in Human Performance were cited as support for the new model. It was also shown that experts in many field showed power functions when learning trials were plotted against performance measures.

Cognitive neuroscience

This special issue of Neuroimage deals with a still further enlargement of the theoretical domains brought to bear on the applications in the workplace. Parasuraman (2011) defines neuroergonomics as the study of the human brain in relation to the performance at work and other everyday life settings. In addition, this collection includes both review and research papers designed to familiarize the reader with the many forms of neuroimaging, including fMRI and EEG, and how they may be applied to diverse issues related to human performance at work. Below I trace the background to this effort to apply neuroimaging to the work environment.

The modern era of neuroimaging began with an effort in the late 1980s to understand how mental operations are localized in the human brain. The study of mental operations had been an important topic in cognitive psychology and cognitive engineering since its inception in the late 1950s (Posner and Raichle, 1994). Newell and Simon (1972) had proposed that computer programs based on think aloud protocols of human experts could serve as models of the human thought processes. In the study of mental imagery (Kosslyn, 1980, 1994), reading (Posner and Raichle, 1994), and mental arithmetic (Dehaene, 1996), mental operations had been isolated experimentally by the time they took to execute. In some cases the operations occurred serially and one could use additive factor methods to determine which variable influenced which process (Sternberg, 1969, 2004). In other cases they occurred in parallel and could be isolated by independently manipulating the times involved by separate variables. For example, when people were asked to compare two words (e.g. DOG, dog) that had the same name but differed physically, it took them about 80 ms longer than if they were physically identical (dog, dog). Changes in the luminance or color of the pair influenced the time for identity but not name matches, while increasing the number of letter names in memory influenced the name match but not physical match times (Posner, 1978).

The initial step in application of neuroimaging to cognitive issues used positron emission tomography (PET) and examined listening and reading individual words (Petersen et al., 1987). When people read words aloud the major activity was in motor areas and in the left anterior insula, but when they produce a word that was the use of the noun they activated left anterior frontal, cingulate and cerebellar areas as well as a posterior temporal–parietal area. The highly automated task of reading produced one set of areas, while when a novel association was required, a new set of areas was activated. During naming novel associations the anterior cingulate was involved in attention to the task, while the left frontal area held the stimulus in mind and Wernicke’s area activation produced the associated meaning. If the same list of words was repeated and participants made the same association the strength of the activations related to the novel association decreased and the set of activations became similar to reading the word aloud (Raichle et al., 1994). If a new list of words was then used all the activations originally found returned. A few minutes of practice lead to the automation of the associations so they were made more reliably and faster and the brain pathway was as though the association was as directly connected to the visual word as it was in reading the word itself. These findings supported the isolation of mental operations in the brain and showed how well they adapted to learning.

fMRI

A major development in 1990 was the use of magnetic resonance to measure localized changes in blood oxygen as a means of mapping brain activity non-invasively (Ogawa et al., 1990). Much subsequent work has confirmed and elaborated the meaning of each of the activations found with PET to the skill of reading words. Two important posterior brain areas operate automatically in the skilled reader. These are in the left fusiform gyrus and the left temporal parietal lobe. The first of these two areas, called the visual word form area, is still somewhat controversial (McCandliss, et al., 2003; Price and Devlin, 2003), but appears to be involved in chunking visual letters into a unit. The visual word form area seems to be abstract in that it can match both upper and lower case versions of the word. A second area is closer to the auditory system and appears to represent the sound of the visual word. These two areas operated automatically in skilled readers of English and they do not seem to work well in children having difficulty in learning to read (Shaywitz, 2003).

fMRI is one of the most frequently used neuroimaging methods in cognitive neuroscience. Despite some drawbacks, such as its lack of portability, low temporal resolution, and relatively high cost, fMRI has also been useful in neuroergonomic studies, as illustrated by some of the papers in this special issue. These include studies of everyday activities and tasks, such as identifying and understanding the movements and actions of other people (Grafton and Tipper, 2012-this issue; Thompson and Parasuraman, 2012-this issue), making complex decisions (Parasuraman, 2011), driving (Calhoun and Pearlson, 2012-this issue), and monitoring and controlling air traffic (Ayaz et al., 2012-this issue).

Connectivity

Neural areas found active in studies of functional anatomy must be orchestrated in carrying out any real task. An approach to studying this connectivity uses fMRI to study the time course of activity and the correlations between active areas. Below we illustrate these methods by primarily considering the connectivity of the anterior cingulate during tasks that involve attention. The anterior cingulate is one area found active during the reading and listening tasks described in the last section. This area of the brain is one that has large-scale connectivity to many other brain areas and is ideally situated to exercise executive control over other brain networks (Posner, 2008).

The executive attention network is involved in the ability to resolve conflict among the many active brain networks competing for the control of behavior. The anterior cingulate is part of a network that includes other important brain areas. According to Bush et al. (2000), an analysis of a number of conflict tasks shows that the more dorsal part of the anterior cingulate is involved in the regulation of cognitive tasks, while the more ventral part of the cingulate is involved in regulation of emotion. The dorsal part of the anterior cingulate has strong connections to frontal and parietal areas involved in cognitive processes. During task performance it establishes contact with brain areas involved in processing information. In one study, for example, participants were asked to select either visual or auditory information. During the selection of visual information the dorsal cingulate showed correlation with visual brain areas and these correlations switched to auditory areas when they were chosen (Crottaz-Herbette and Menon, 2006). When participants process emotional information the more ventral parts of the cingulate are active and connected to limbic areas related to the emotion (Etkin et al., 2006). Such correlational approaches to identifying the connectivity of different brain regions are clearly relevant to neuroergonomic studies examining the brain networks associated with common everyday tasks. One example from the special issue involves the use of independent component analysis (ICA) to identify the functional connectivity of brain regions activated during different phases of simulated driving, and how these networks are affected while driving under the influence of alcohol (Calhoun and Pearlson, 2012-this issue). A second example concerns the effects of different types of emphasis training (Gopher et al., 1989) when playing a complex videogame on the functional connectivity of brain attentional networks (Voss et al., 2012-this issue).

Another approach to the measurement of connectivity involves the measurement of fiber tracts that connect neural areas by use of diffusion tensor imaging (DTI) to view noninvasively the white matter connections between brain areas. This form of imaging uses the diffusion of water molecules in particular directions due to the presence of myelinated fibers (Conturo et al., 1999). Thus this method provides a way of examining the physical connections present in the brains of people and allows tracing of fiber pathways during different stages of human development or as a function of experience and training.

Since fMRI is noninvasive it is possible to use multiple scans to examine changes that occur with learning and development (Kelly and Garavan, 2005). This is obviously an important tool for neuroergonomic applications. It is common for learning on a task to decrease the number and amount of activation of the associated brain networks. The rate of these changes may vary from milliseconds to years depending upon what is being learned. Connectivity of the network can also be enhanced by practice (McNamara et al., 2007). A major development in this field is recording brain activity at rest to determine the relative connections between brain areas. This method allows studies at many ages since no task is needed. Studies of changes in connectivity with development show that local connections dominant in children give way to longer connections more prominent in adults (Fair et al., 2009). This change in connectivity is often accompanied by reduced number and size of activations, just as is found for practice in a given task (Savoy, 2001).

Because of the relatively long delays between input and peak BOLD fMRI signal, small time differences may be hard to detect. Recently Raichle (2010) has identified the BOLD response measured in fMRI with slow wave potentials found in EEG. However, the use of event related electrical and magnetic signals allow recording of much faster brain activity. The approach to the examination of temporal connections between brain areas based on electrical or magnetic signals can give higher temporal resolution, and can be combined with MRI to improve their spatial localization.

Event related potentials

When a signal is presented many times the electrical or magnetic activity can be averaged to form an event related potential (ERP) that indicates the activity following at each few millisecond after presentation. Dehaene (1996) used electrical recording from scalp electrodes to map out the time course of mental activity involved in the task of determining whether a visual digit was above or below 5. He averaged ERPs following the presentation of a single digit that was to be classified as above or below 5. The first 100 ms involved activity in the visual system. When the input was an Arabic digit both hemispheres were active, but when it was a spelled digit (e.g. six) activity was in the visual word form system. In the next 100 ms differences were found between digits close to and far from five. This difference was in electrodes over the parietal brain areas known to be involved in representing the number line. Before output there was activity in motor areas and following a trial in which the person made an error there was an error related negativity in electrodes over the frontal midline localized to the anterior cingulate. Although recognition of the quantity of a digit is a very elementary aspect of numeracy, training in the appreciation of the value of a number has been shown to be an important contributor to success in elementary school arithmetic (Griffin et al., 1995).

Oscillations

The complex electrical signal coming from scalp electrodes can be decomposed into sine and cosine waves by Fourier analysis. There is a great deal of interest in the functions of oscillations both in changes of brain state and in integrating brain activity in different brain systems. During sleep deep slow waves predominate and in the awake resting state, created by closing the eyes, alpha frequency (about 10 Hz) dominates particularly over posterior electrodes. Following an error made in response to a task, which is detected by the person, there is activity in the theta band (3 Hz) (Berger et al., 2006). It has been hypothesized that high frequency gamma activity (40 Hz) or higher is important in order to tie together distant brain regions that are analyzing a single object (Womelsdorf et al., 2007).

Electrical and magnetic recordings of brain activity have long been used in ergonomics and human factors (Kramer and Parasuraman, 2007). This usage is also reflected in many papers in the special issue. These studies used ERPs and EEG to investigate such issues as biological motion and action understanding (Grafton and Tipper, 2012-this issue; Thompson and Parasuraman, 2011), affective processing (Parasuraman and Jiang, 2012-this issue), and group decision-making, or collective wisdom (Eckstein et al., 2012-this issue).

Lesions

The activation of brain networks does not mean that all parts of the network are needed to carry out the task. In the past, effects of brain lesions have been a primary way to indicate brain areas which, when lost, will prevent the persons from carrying out certain tasks. For example, damage to areas of the right parietal lobe has led to the neglect of the left side of space in multiple sensory systems.

A good example of the use of lesion data in conjunction with imaging is in a study of a patient who following a stroke was unable to read words when they were presented to the left of where he was currently looking (fixation) but could read them fluently when presented to the right of fixation (Cohen et al., 2004). Imaging showed that there was interruption of the fibers that conducted information to the visual word form area from the occipital lobe of the right hemisphere. When words were presented to the left of fixation (i.e., directly to the right hemisphere), the patient could only sound them out letter by letter although he clearly maintained all the reading skills as evidenced by his performance with words presented to the right visual field (i.e., directly to the left hemisphere), so that they did reach the visual word form area. This study shows clearly that the visual word form area is a necessary condition for fluent reading.

It is now possible to use brief magnetic pulses applied to the scalp overlying the brain area of interest (transcranial magnetic stimulation, TMS) to disrupt parts of the network at particularly times to observe its influence on task performance. One striking example of this technology shows that readers of Braille use the visual system. When TMS was applied to visual cortex Braille readers had a specific problem in reading words through touch, suggesting that the visual system was used to handle spatial aspects of the tactile input in Braille (Pascale-Leone and Hamilton, 2001).

Lesion data can be used to confirm and extend theories arising from imaging techniques. While educators are not usually confronted with patients with specific brain lesions due to stroke, findings from these patients can often illuminate specific learning difficulty, dyslexia or dyscalculia that arise in development.

Individual differences in network efficiency

Neuroimaging has provided a new perspective on the nature of individual differences. Most of cognitive psychology was concerned with averaged mental performance while the study of individual differences in mental ability was handled in a separate area often called psychometrics. Although most of the networks studied by neuroimaging are common to all people there are differences in the efficiency with which the networks operate, as also described by Miller et al. (this issue) in a paper in this special issue. Some of these differences may be due to genetic variations among people but the expression of these genetic variations is also influenced by experience. Genes code for different proteins that influence the efficiency with which modulators such as dopamine are produced and/or bind to their receptors. These modulators are in turn related to individual difference in the efficiency of the brain networks. The paper by Parasuraman and Jiang (2012-this issue) in the special issue describes the role of genetic factors in inter-individual variation in networks associated with decision-making and affective processing. There is a great deal in common among humans in the anatomy of high-level networks, and this must have a basis within the human genome. The same genes that are related to individual differences are also likely to be important in the development of the networks that are common to all humans. Learning can build upon preexisting brain networks to achieve new functions. For example, primitive appreciation of number is present in infancy. However when used together with language networks they can form a basis for numerical calculation (Dehaene and Cohen, 2007).

In the study of attention individual differences have been linked to differences in genetic variation. The association of an attention network with the neuromodulator dopamine is a way of searching for candidate genes that might relate to the efficiency of the network. For example, several studies employing conflict related tasks found that alleles of the catecholo-methyl transferase (COMT) gene were related to the ability to resolve conflict. A number of other dopamine genes have also been proven related to this form of attention. In addition, research has suggested that genes related to the cholinergic system influence orienting of attention (Parasuraman et al., 2005; see Green et al., 2008 for a review).

It was also possible to show that some of these genetic differences influenced the degree to which the anterior cingulate was activated during task performance in studies using brain imaging (Fan et al., 2003). In the future it may be possible to relate genes to specific nodes within neural networks, allowing a much more detailed understanding of the origins of brain networks.

While genes are important for common neural networks and individual differences in efficiency there is also an important role for specific experiences. For example, several genes including the DRD4 gene and the COMT gene have both shown to interact with aspects of quality of parenting. This provides evidence that aspects of the social situation in which children are raised can influence the way in which genes shape neural networks influencing child behavior (Posner et al., 2007).

Learning and training

If brain network efficiency is influenced by parenting and other cultural influences it should be possible to develop specific training methods that can be used to influence underlying brain networks. For example, one study tested the effect of training during the period of major development of executive attention, which takes place between 4 and 7 years of age. An improvement in the brain underwork underlying was found in trained children, along with generalization to other aspects of cognition (Rothbart et al., 2009). Similar studies have shown improvement of attention in classrooms following training in working memory (Klingberg, 2012). We (Tang and Posner, 2009) have distinguished between attention training through practice on particular networks and attention state training, by training a brain state that influences attention and other functions. Using a form of meditation we have been able to show changes in brain activation and connectivity following 3 to 11 hours of training (Tang et al., 2010).

The wartime work by Paul Fitts with pilots attested to the importance of training to improve flight performance and reduce errors. Training continues to be an important facet of ergonomics today, and its importance is reflected in several papers in the special issue that examine different aspects of training. These include the effects of videogame training on connectivity of brain attentional networks (Voss et al., 2012-this issue) and the influence of transcranial direct current brain stimulation on learning and perceptual performance (Clark et al., 2012-this issue).

Applications to work

In this brief commentary I have used the example of processing of words to illustrate some of the important steps in the development of neuroimaging. The same general methods can be applied to most real life tasks, for example to the issues raised by Parasuraman (2011) in describing neuroergonomics and the many illustrations included in this special issue. Many work tasks have been analyzed in terms of the subroutines or computations that are necessary for a computer to simulate human performance. These methods connect current work involved in imaging with the earlier efforts to apply psychology to the workplace such as cognitive engineering and human information processing. The use of brain imaging methods has great potential to supplement studies of human information processing and cognitive engineering in application to the workplace. However, neuroimaging methods will be used more effectively if the background in the information processing and cognitive engineering approaches to Ergonomics are kept in mind when trying to integrate these new methods and ideas from brain research.