The Role of Expertise Research and Human Factors in Capturing, Explaining, and Producing Superior Performance

Abstract

Objectives

The goal of this article is to identify some of the major trends and findings in expertise research and their connections to human factors.

Background

Progress in the study of superior human performance has come from improved methods of measuring expertise and the development of better tools for revealing the mechanisms that support expert performance, such as protocol analysis and eye tracking.

Methods

We review some of the challenges of capturing superior human performance in the laboratory and the means by which the expert performance approach may overcome such challenges. We then discuss applications of the expert performance approach to a handful of domains that have long been of interest to human factors researchers.

Results

Experts depend heavily on domain-specific knowledge for superior performance, and such knowledge enables the expert to anticipate and prepare for future actions more efficiently. Training programs designed to focus learners’ attention on task-related knowledge and skills critical to expert performance have shown promise in facilitating skill acquisition among nonexperts and in reducing errors by experts on representative tasks.

Conclusions

Although significant challenges remain, there is encouraging progress in domains such as sports, aviation, and medicine in understanding some of the mechanisms underlying human expertise and in structuring training and tools to improve skilled performance.

Applications

Knowledge engineering techniques can capture expert knowledge and preserve it for organizations and for the development of expert systems. Understanding the mechanisms that underlie expert performance may provide insights into the structuring of better training programs for improving skill and in designing systems to support professional expertise.

Experts are sometimes defined as outliers from the general population: those more than two standard deviations above the mean, showing consistently superior performance on representative tasks from their domain (Ericsson & Charness, 1994). Given that guidelines for design typically try to encompass the middle 95% of the population, it seems strange for the field of human factors to be concerned with expertise. However, experts may contribute disproportionately to the welfare of their societies (Charness & Krampe, 2008). Consider an expert fighter pilot navigating a $100 million aircraft trying to prevent an opponent from destroying a $5 billion aircraft carrier. A design change that provides even a minor advantage to this skilled pilot could prove highly cost-effective.

Some applied issues addressed by contemporary expert performance researchers were raised by the progenitors of human factors. Fitts (1947) attempted to capture pilots’ eye movements during landings and retrospective reports of critical aviation incidents, and Gagne (1954) noted some critical hazards of using poor criterion measures in the evaluation of simulation fidelity.

Common to human factors and expertise research is the admonition to “know thy user.” Progress in expertise research (see Ericsson, Charness, Hoffman, & Feltovich, 2006) has resulted mainly from improved methods for measuring expertise and the development of tools that reveal the mechanisms that support expert performance.

There are many approaches to the study of expertise (for an experience approach, see Ericsson et al., 2006; Salthouse, 1990). We restrict ourselves here to an influential one known as the expert performance approach. Its general premise is that any claim regarding expertise requires empirical observations of superior, reproducible task performance. (Relying on someone’s title or years of experience can be inadequate.) Thus, modern researchers begin by constructing representative tasks on which superior performance can be directly observed. In some cases, the representative task can be sampled directly from real-world situations (e.g., best move selection from a high-level, unfamiliar chess position). However, alterations may be necessary to re-create the task in a laboratory setting by constructing simulations or scenarios depicting realistic critical incidents (Ericsson & Williams, 2007).

Once superior performance has been captured under controlled conditions, experiments can be designed to identify the underpinnings of superior performance. Process-tracing techniques such as the elicitation of think-aloud reports (Ericsson, 2006) or the collection of eye movement data (e.g., Reingold,Charness,Pomplun,&Stampe,2001) can reveal the mechanisms supporting expertise.

An important caveat for the expert performance approach – and, indeed, for any approach that uses research designs where people are not randomly assigned to individual difference levels (e.g., age, gender, education level) – is that studies are quasi-experimental and hence cannot unambiguously determine causation for mechanisms identified as supporting expert performance. Ultimately, if the expert performance approach has validity, it should be demonstrable through the development of skill-sensitive training procedures to help bring novices up to high levels of performance more quickly. Some successful applications can be seen in training apprentices (Kalyuga, Chandler, & Sweller,1998) and training vehicle threat detection (Kirlik, Walker, Fisk, & Nagel, 1996). Also, by identifying the mechanisms for expert performance, such research may pinpoint ways to design better tools to support the expert. Examples would be the case of performing information search tasks (Hollands & Merikle, 1987), selecting input devices (Sutter & Ziefle, 2005), and using sound to improve patient monitoring by anesthesiologists (Watson & Sanderson, 2004).

A classic finding from the expertise area, dating back to the earliest work on telegraphy (Bryan & Harter, 1899) and through work on chess (Chase & Simon, 1973), is that the highly efficient performance of the expert depends on acquiring domain-specific knowledge rather than on a general ability advantage. Evidence for this proposition is the skill-by-task interaction observed in many domains, usually with memory and perception tasks. Experts show far superior perception and recall performance on briefly presented domain-related materials (e.g., structured chess positions) but differ minimally or not at all from novices in performance on recall of domain-unrelated materials (e.g., quasi-random chess positions). This perceptual and knowledge advantage enables the expert to anticipate and prepare for future actions in sports (Williams, Ward, Knowles, & Smeeton, 2002), games (Jastrzembski, Charness, & Vasyukova, 2006), piloting aircraft (Sohn & Doane, 2004), and optimizing industrial production processes (Prietula, Feltovich, & Marchak, 2000).

Given that domain knowledge is regarded as critical to expert performance, knowledge engineering techniques have been developed to mine the knowledge of experts (Hoffman & Lintern, 2006; Hoffman, Crandall, & Shadbolt, 1998), with the aim of preserving it for organizations when experts depart and to create computer-based expert systems (e.g., Shortliffe, 1976).

Researchers employing the expert performance approach have reported a number of consistent findings that have implications for applied human performance research. Foremost among the findings is that many experts report extended training and/or apprenticeship (a decade or longer) prior to their attainment of superior performance (Ericsson & Lehmann, 1996). One explanatory mechanism proposed to account for the development of skill during this period is the notion of deliberate practice – challenging activities designed to target and correct specific weaknesses in performance skills (Ericsson, Krampe, & Tesch-Römer, 1993). For instance, skilled musicians frequently practice technical études focusing on particular dimensions of performance such as articulation or dynamics (Gruson, 1988). In domains that are less motor skill oriented, such as chess, players may improve their tactical skills by studying experts’ previous games, following one move at a time while attempting to predict the subsequent moves (Charness, Tuffiash, Krampe, Reingold, & Vasyukova, 2005).

It is often difficult to differentiate deliberate practice activities from other forms of practice, so in some applied domains, experiences linked with deliberate practice are used as surrogates. For instance, flying hours on a given military aircraft (vs. hours on all aircraft) were taken as a proxy for deliberate practice when examining how effectively pilots coped with mechanical failures in flight (McKinney & Davis, 2003).

The consequences of extended engagement in deliberate practice are quantitative and qualitative changes in cognitive representations of task-relevant knowledge. For instance, expert soccer goalkeepers learn to attend to particular postural characteristics and body angles to anticipate the direction of an oncoming kick (Williams et al., 2002). In other motor-intensive performance domains, such as piano playing and typing, experts demonstrate a greater eye-hand span that enables them to program overlapping keystrokes (Furneaux & Land, 1999; Salthouse, 1986).

In predominantly cognitive domains, such as competitive memory performance, memorists learn to chunk incoming stimuli into meaningful units and employ hierarchical retrieval mechanisms during test performance (Chase & Ericsson, 1982; Ericsson, Delaney, Weaver, & Mahadevan, 2004). Experts in other domains with high problem-solving and reasoning demands, such as physics and medicine, are more likely to identify the essential principles underlying particular cases than domain novices, whose representations tend to gravitate toward surface features. The superior problem representation of experts permits more effective search strategies, such as forward search through the problem space (Chi, Feltovich, & Glaser, 1981; de Groot, 1978; Patel & Groen, 1986). We now examine in more detail a few representative domains characterized by robust measurement of expertise and substantial progress in identifying underlying mechanisms.

SPORTS

Recent studies have demonstrated that key psychological and physical elements of sport performance can be observed reliably under controlled conditions, with further studies demonstrating how the information gained from such observations can be used to improve and refine critical performance skills (Ward, Williams, & Hancock, 2006).

Williams et al. (2002) videotaped expert and novice tennis players during serve receive, noting differences in their time to react and accuracy of returns. Interventions using expert modeling and feedback produced significant increases in serve receive performance (Farrow, Chivers, Hardingham, & Sachse, 1998; Smeeton, Williams, Hodges, & Ward, 2005). Analogous programs for field hockey and soccer goalkeepers have also demonstrated positive results (for a review, see Ward et al., 2006).

Fadde (2007) described a number of studies on the recognition of pitches by batters in baseball. Citing prior laboratory observations of eye movements by expert and novice baseball players (Paull & Glencross, 1997), Fadde noted that the locus of superior pitch-tracking skills in expert baseball batters could be traced to their superior recognition of cues extracted from perception of the pitcher’s arm motion and release, as well as the first third of the trajectory of the ball during flight. Consequently, Fadde (2006) designed a video observation system to guide players’ attention to these critical cues, followed by a controlled study of its effects on a sample of collegiate baseball players. Fadde demonstrated real-world transfer of the video training program, reporting a significant increase in batting performance of the trained players during competitive play relative to a nontrained control group equated for pretraining hitting ability.

AVIATION

During World War II, Fitts (1947) suggested that many crash landings among military pilots were caused by poorly designed instrument panels, with further observational work using crude measures of eye movements suggesting significant variability in the frequency of fixations on cockpit equipment during landings (Fitts, Jones, & Milton, 1950). Using modern eye-tracking equipment capable of producing more reliable measures of scan paths, contemporary researchers in a number of labs (Bellenkes, Wickens, & Kramer, 1997) have observed systematic differences between more and less skilled pilots in the direction of gaze between instruments and the external view during landings. A number of subsequent trials of computer programs have demonstrated the utility of training new pilots to follow the same type of instrument viewing patterns demonstrated by experts in these laboratory studies (Gopher, Weill, & Bareket, 1994), with a significant drop in errors reported during both simulated and actual flights.

Another area of considerable concern in aviation is weather-related decision making. Longitudinal records over the past half-century indicate adverse weather as the third leading cause of aviation incidents after pilot error and equipment malfunctions, as well as a contributing factor in roughly 20% of all accidents (National Transportation Safety Board, 2005). Researchers have found individual differences in pilots’ abilities to recognize adverse weather conditions, with more experienced pilots generally making better judgments. Until recently, however, it was not known how such judgments were being made.

By employing think-aloud reports, Wiggins and O’Hare (1995) demonstrated that highly experienced pilots focused on a relatively small set of weather parameters and weighted them in a different manner than less experienced pilots (Rockwell & McCoy, 1988). Wiggins and O’Hare (2003) subsequently designed and tested a cue-based training approach for recognition of poor weather conditions. The researchers demonstrated limited transfer, inasmuch as the pilots receiving the cue training made better weather decisions during a simulated flight. Whether such training results in a significant decrease in accidents among those exposed to it has proven difficult to assess. Nevertheless, the application created by these authors is now recommended by the Federal Aviation Administration (FAA) as a part of pilot training (Jensen & Hunter, 2002).

MEDICINE

Like pilots, medical professionals have also benefited from the synergy of expert performance and human factors research. One area in which this is evident is in the design of health information and decision support systems. Medical practitioners, particularly in intensive care settings, are required to make rapid and accurate decisions about the health of their patients. However, medical technologists have introduced a dizzying array of computerized recordkeeping systems and monitoring devices into patient care, and failures to properly use or interpret the data from such devices may be a significant factor in medical treatment errors (Koppel et al., 2005; Walsh & Beatty, 2002).

Seeking to minimize these types of problems, Alberdi et al. (2001) observed senior- and junior-level physicians and nurses in a neonatal intensive care unit and found that senior doctors and nurses used computer physiological monitoring equipment both more frequently and in a different manner than the junior-level staff. These observations, as well as informal feedback interviews with the same staff, suggested modifications to the settings of monitoring equipment displays, along with a number of training interventions to improve signal interpretations. Subsequent trials by a companion research group (Hunter et al. 2003) demonstrated the utility of designing neonatal monitoring software according to actual observations of staff usage.

Simulators have become an integral part of both the training and assessment of medical personnel, particularly in high-risk areas such as surgery and anesthesia (Kneebone, 2003). In addition to allowing repeated, rapid exposure to critical events that might otherwise occur very infrequently in the actual environment, medical simulators have also provided health care administrators and researchers with the tools to assess professional skills without endangering the lives or well-being of actual patients. A number of studies have been conducted in which highly experienced medical staff have demonstrated superior performance using such simulators, with eye-tracking data and verbal reports elucidating some of the critical mechanisms that may underlie such performance (Ericsson, 2004).

However, some recent work has also pointed out the potential drawbacks of medical simulators. Moorthy, Munz, Adams, Pandey, and Darzi (2005) found significant differentiation between junior and intermediate trainees on many technical elements of a simulated vein operation but not between junior and senior trainees. Moorthy et al. also noted failures of simulators to clearly identify and isolate errors in team communication between the primary surgeon, the surgeon’s assistant, and the anesthesiologist. This, as well as other work (Maran & Glavin, 2003), suggests that different kinds of simulators may be optimal for measuring particular surgical skills at different overall levels of surgical expertise and also serves to highlight the broader importance of adapting simulator design to the skills and needs of the targeted user and the constraints placed on that user by the particular situation being simulated.

THE COSTS AND BENEFITS OF EXPERTISE APPROACHES

We have highlighted the positive impact of integrating the methods and principles of expert performance research with those of human factors. However, as with any empirical approach, there are drawbacks. Because experts are by definition a rarity within the population, it can be costly to identify and recruit them for laboratory research. In domains such as military aviation, emergency medicine, or professional sports, the limited availability of top-level personnel to participate in lengthy laboratory studies may present a significant obstacle. Also, given that there will always be variability in performance, as well as in the contents of process-tracing data (e.g., think-aloud reports), one cannot always be confident in generalizing from observations collected from a few experts.

Furthermore, it can be costly to make the necessary links between cross-sectional studies that identify skill differences (via novice-expert comparisons) and training studies that can be used to verify that the observed differences are essential to expert performance and that the appropriate skills and knowledge structures can be imparted efficiently. Longitudinal training studies may take more time and money than most engineering or usability labs can afford.

Such problems pose significant challenges. The path to progress may require a great deal more collaboration and discourse between researchers following the expert performance approach and those following complementary approaches found elsewhere in the human factors community.

Looking forward, we see a continuing need for integration of the observational tools employed by expert performance researchers (e.g., verbal reports/protocol analysis, eye tracking) into the human factors laboratory. Detailed observations of skilled performers engaged in representative tasks may lead to further refinements of training technologies and hardware/software interfaces that better reflect the abilities brought to bear in complex task environments. In turn, new measurement technologies developed by both hardware and software engineers may permit the development of reliable performance metrics in domains that presently lack objective measures of skill (e.g., Weiss & Shanteau, 2003). Despite the difficulty in pursuing research with experts, the studies undertaken in domains such as sports, medicine, and aviation during the past decade show the promise of the expert performance approach.

Biography

Neil Charness is the William G. Chase Professor of Psychology and an associate of the Pepper Institute on Aging and Public Policy at Florida State University. He obtained his Ph.D. in psychology from Carnegie Mellon University in 1974.

Michael Tuffiash is a doctoral student in the Psychology Department at Florida State University, where he received his M.Sc. in psychology in 2002.