Prospective memory in an air traffic control simulation: External aids that signal when to act

Abstract

At work and in our personal life we often need to remember to perform intended actions at some point in the future, referred to as Prospective Memory. Individuals sometimes forget to perform intentions in safety-critical work contexts. Holding intentions can also interfere with ongoing tasks. We applied theories and methods from the experimental literature to test the effectiveness of external aids in reducing prospective memory error and costs to ongoing tasks in an air traffic control simulation. Participants were trained to accept and hand-off aircraft, and to detect aircraft conflicts. For the prospective memory task participants were required to substitute alternative actions for routine actions when accepting target aircraft. Across two experiments, external display aids were provided that presented the details of target aircraft and associated intended actions. We predicted that aids would only be effective if they provided information that was diagnostic of target occurrence and in this study we examined the utility of aids that directly cued participants when to allocate attention to the prospective memory task. When aids were set to flash when the prospective memory target aircraft needed to be accepted, prospective memory error and costs to ongoing tasks of aircraft acceptance and conflict detection were reduced. In contrast, aids that did not alert participants specifically when the target aircraft were present provided no advantage compared to when no aids we used. These findings have practical implications for the potential relative utility of automated external aids for occupations where individuals monitor multi-item dynamic displays.

Examining the Effectiveness of External Aids that Signal When to Perform a Prospective Memory Task in an Air Traffic Control Simulation

Many activities at work and in our personal life depend on our ability to remember to perform tasks at appropriate points in the future. This is referred to as Prospective Memory (PM). Individuals are often engaged in other activities during the interval between planning and performing intentions. Furthermore in order to execute intended actions, individuals must often interrupt other ongoing activity. In most cases, there are no external agents directing individuals to engage in a memory search at the point that the PM task should be performed and individuals need to self-initiate the retrieval of the intentions. In aviation, incident analyses and interview data indicate that pilots and air traffic controllers sometimes fail to complete intentions (Dismukes, 2008; Shorrock, 2005). For instance, recently two commercial airline pilots flew approximately 100 miles past their destination because they failed to remember to interrupt other ongoing activity to begin their descent (Maynard & Wald, 2009). One way to minimize human error is to provide operators with automated support tools. In the current article, we develop and test the effectiveness of external aids for reducing PM error and interference to ongoing tasks in an Air Traffic Control (ATC) simulation.

The ATC task we employ simulates work domains where individuals continuously monitor multi-item dynamic displays, such as ATC, naval radar tracking, and air battle management. The ATC task involves various ongoing activities including detecting conflicts, and accepting and handing off aircraft. A PM task of remembering to deviate from a routine acceptance procedure when a target aircraft is encountered is embedded within these ongoing tasks. Loft and Remington (2010) demonstrated that individuals were slower to perform ongoing tasks when they held intentions to deviate from routine in this ATC simulation, an effect referred to as ‘the cost to the ongoing task’ or ‘task interference’ (Hicks, Marsh, & Cook, 2005; Smith, 2003). This cost indicates that attention was allocated to the PM task at the expense of ongoing tasks. In ATC and other safety critical work contexts, such costs to ongoing tasks would be unacceptable because they would decrease the capacity of operators to manage task demands (Loft, Sanderson, Neal, & Moiij, 2007; Sperandio, 1971). Loft and Remington also found that, despite costs to ongoing tasks, participants often did not remember to perform intentions, and concluded that individuals had difficulty maintaining their intent to monitor the task environment in order to determine when deviation from routine was required (Norman, 1981; Reason, 1990).

With this in mind, our goal was to develop external aids that would allow individuals to allocate attention to PM tasks more precisely, thereby decreasing PM error and costs to ongoing tasks. In developing our aids, we consulted theories in the basic experimental literature that describe how individuals maintain and retrieve intentions (Einstein et al., 2005; Marsh, Cook, & Hicks, 2006; Smith, 2003, 2008), as well as theories that delineate how to design automated tools to capture and direct attention to infrequent task events in dynamic displays (Sheridan & Parasuraman, 2006; Wickens & Rose, 2001; Yantis, 1993).

Prospective Memory and External Aids in Air Traffic Control

ATC is a prototypical example of an occupation where individuals must remember to perform intended actions whilst continuously monitoring a dynamic display (for a recent review of human factors issues in ATC, see Durso & Manning, 2009). Air traffic controllers perform sets of routine activities to expedite and preserve orderly traffic flows and maintain aircraft separation. The radar screen displays a sector consisting of sector boundaries and flight paths. Aircraft are indicated by symbols with data blocks attached that contain aircraft information such as call sign, altitude, airspeed, and type. The positions of aircraft symbols and their data blocks continuously update to track the aircraft’s progress through the sector. The highest priority of the controller is to detect and resolve conflicts. Aircraft are in conflict if they will violate both lateral (e.g., 5 nautical miles [nm]) and vertical (e.g., 1,000 feet [304.8 m]) separation standards simultaneously. The controllers’ second goal is to ensure aircraft reach destinations efficiently.

Controllers often cannot perform intended actions immediately because of their workload or because the current situation does not permit it (Loft et al., 2007). For example, a pilot may verbally request a new altitude due to weather. If the controller is busy with other tasks, or the aircraft is not currently under jurisdiction, s/he may need to take mental note to come back and fulfill this request when accepting the aircraft into the sector. In these situations controllers sometimes fail to remember to perform an atypical intended action, substituting a more routine action instead (Dismukes, 2008; Reason, 1990). Controllers can use external aids to reduce the likelihood of such errors, including writing intentions on notepads or ‘post-it’ notes, typing intentions in free text boxes on ATC displays, or highlighting aircraft (Neal & Moiij, 2008). Despite the use of such aids, Shorrock (2005) reported that 38% of memory errors in ATC in the United Kingdom involved situations where controllers forgot to perform intended actions.

To our knowledge, only one study has tested whether external aids can facilitate PM in ATC (Vortac, Edwards, & Manning, 1995). Participants in the Vortac et al. study were required to direct aircraft to routine destinations and keep aircraft separated, but were sometimes asked to direct aircraft to alternative instead of routine destinations. Participants were more likely to remember to do this when PM instructions were displayed on electronic flight strips (e.g., “[call sign] AAL123 go to [destination] X”) at the time intentions needed to be performed. The positive effects of external aids (reminders) on PM performance have also been reported in the basic literature (Finstad, Bink, McDaniel, & Einstein, 2006; Guynn, McDaniel, & Einstein, 1998).

Despite the provision of external aids, Vortac et al. (1995) reported that participants still failed to deviate from routine on up to 20% of occasions. Ideally, PM performance in contexts such as ATC should be error free. In addition, Vortac et al. did not examine whether external aids were effective in reducing the costs to ongoing tasks reported in both applied (Loft & Remington, 2010) and basic (Smith, 2003) PM tasks. In fact, the potential benefit of external aids in reducing costs to ongoing tasks has not previously been considered by the PM literature. This is a crucial omission because even slight delays in task completion would be unacceptable in some safety-critical work contexts. In the sections below, we use theory to build upon the work of Vortac et al. and develop external aids for reducing PM error and costs to ongoing tasks in an ATC simulation context.

Theoretical Approach

There are several theories that posit how intentions are retrieved when target events occur in ongoing tasks. The Multiprocess view (MPV; Einstein et al., 2005; McDaniel & Einstein, 2007) proposes that under specialized conditions intentions can be automatically retrieved. However, the MPV also proposes that in many circumstances individuals engage in monitoring processes in order to recognize targets. While there is debate over the question of whether intentions can be automatically retrieved (Einstein & McDaniel, 2010; Smith, 2010), the preparatory attentional and memory processes theory (PAM; Smith, 2003), as well as other theories (Burgess & Shallice, 1997; Guynn, 2003), agree with the MPV that individuals do often explicitly check ongoing tasks for PM targets. The standard line of evidence that these monitoring processes have been engaged are performance decrements on ongoing tasks when individuals hold intentions versus when the ongoing task is performed alone (see Smith, Hunt, McVay, & McConnell, 2007, for review). This cost to the ongoing task is obtained on non-target trials that precede the target event and thus does not reflect a response cost associated with executing intended actions; however, it is possible that the cost reflects not just monitoring, but also rehearsal of the intended action or target events, and processes involved in discriminating between target and non-target events (Smith, 2010)¹. In the current paper we adopt the proposal made by the PAM theory that the cost to the ongoing task is due, at least in part, to ‘preparatory attentional processes’ (Smith, 2003, 2008). Preparatory attention can include explicit monitoring processes, but also can involve processes occurring outside the focus of attention (Smith, 2008); nonetheless, the processes will incur a cost, although often subtle, to ongoing tasks (Smith et al., 2007).

Loft and Remington (2010) recently demonstrated that embedding a PM task in an ATC simulation can produce a cost to ongoing tasks. Participants were trained to accept aircraft into sectors and detect conflicts. For the PM task, participants were required to substitute new actions for routine actions when accepting particular aircraft. Participants were slower to detect conflicts and make aircraft acceptance decisions when they held PM intentions compared to when they did not. Loft and Remington concluded that participants were allocating preparatory attention and failed to deviate from routine if they did not maintain this preparatory attention. It is likely that significant amounts of resources were required to inhibit aircraft acceptance routines (Logan, 1988; Shiffrin & Dumais, 1981) in order to detect target aircraft and determine when deviation from routine was required (Norman, 1981; Reason, 1990).

A more effective strategy would be to engage in preparatory attention only when the opportunity to carry out the intended action is likely. For example, if I have an intention to buy medicine on the way home from work, allocating resources to preparing to perform this intention throughout the entire workday would not be effective. Instead, it would be more effective to allocate preparatory resources when the intention can be executed, in this case in the context of driving home. There is evidence that individuals can allocate preparatory attention in a context specific manner (Loft, Kearney & Remington, 2008; Marsh et al., 2006). For example, Marsh et al. (2006) reported reductions in ongoing task cost when participants were informed that stimuli about to be presented in the next block of trials were irrelevant to target detection.

The current article investigates whether PM error and costs to ongoing ATC tasks can be reduced by presenting external aids that correspond to the temporal occurrence of target aircraft, thereby notifying participants when to engage preparatory attention. Designers of human-computer interaction systems have long used what is known about attentional control to create cues that capture and direct attention to important events in dynamic displays (Sheridan & Parasuraman, 2006; Wickens & Rose, 2001). Abrupt onset signals are especially potent in capturing visual attention (Jonides & Yantis, 1988; Yantis, 1993), because they elicit reflexive covert attention shifts without the need for resource demanding voluntary orientating (Jonides, 1981; Posner, 1980). On this basis, we predicted that the most effective external aid for capturing attention should be one that is set to flash (blink) when deviation from routine is required.

Experiment 1: ATC Simulation and Predictions

Participants assumed the role of air traffic controllers. An example of the display screen is presented in Figure 1. Participants detected conflict and handed off aircraft. As aircraft approached the sector they flashed for acceptance. Participants accepted aircraft by clicking in the aircraft symbol (circle) then pressing the A key. As aircraft exited sectors, they flashed for hand-off. Participants handed-off aircraft by clicking in the aircraft symbol then pressing the H key. Conflicts occurred when an aircraft pair simultaneously violated 5 nautical mile lateral and 1000ft vertical separation. Participants were required to detect and prevent conflicts by changing the altitude of aircraft. At test, participants continued to accept and hand-off aircraft, and detect and prevent conflicts. Before some test trials they were also asked to press an alternative key (9 key) rather than the routine key (A key) when accepting target aircraft (e.g., an aircraft with a speed greater than 48). This requirement to remember to deviate from the routine aircraft acceptance procedure when target aircraft were accepted served as the PM task.

Figure 1.

A screenshot of the ATC-lab^Advanced program. Inbound aircraft were black (e.g., aircraft C66) as they approached the sector, flashing orange for acceptance (C57) when they reached within 5 miles of the sector boundary. Aircraft turned green (C46) when accepted. When outbound aircraft crossed the sector boundary they flashed blue (C26), and then turned white (C30) when handed off. Aircraft were in conflict if they were traveling at the same altitude and would violate the minimum lateral separation of 5 nm in the future. The example in Figure 1 shows that the individual had changed the altitude of C55 from 340 to 300 in order to avoid a conflict with C37. In the top right corner, C62 is in conflict with C93 (i.e., they will violate separation in the future if not intervened). In this example the external aid is set to remind participants they need to press the 9 key for aircraft with speed greater than 48. Aircraft C57 has a speed greater than 48 and is currently flashing for acceptance. Under flash aid conditions, the external aid would also flash orange when C57 flashes orange for acceptance. The running score (135 points) is presented in the middle right hand side of the display.

We manipulated four conditions within subjects. Under control conditions, participants were not given a PM instruction and the routine acceptance response was to be made for all aircraft. Under no aid conditions, participants were given PM instructions, but were not provided an external aid. Under static aid conditions, participants were given PM instructions and were provided an aid (see Figure 1) that specified the PM target aircraft and intended action. This aid remained on the display for the entire trial for which the intention was active, even when target aircraft were not present (Vortac et al., 1995). Under flash aid conditions, participants were provided these same aids that specified the target aircraft and intended action. However, in contrast to static aids, the flash aid flashed (blinked) the color orange at the time when a target aircraft was flashing at the sector boundary and needed acceptance.

We used the Loft and Remington (2010) indices of ongoing task cost. They argue that the time taken to accept non-target aircraft after they have been selected for acceptance indicates the extent to which individuals are monitoring (Einstein et al., 2005; Guynn, 2003; Smith, 2003) the PM status of aircraft when deviation from routine may be required. Conflict detection could also be affected by participants checking the PM status of aircraft before assessing the conflict status of aircraft. We also measured aircraft hand-off decision time, which should be less affected by such monitoring mechanisms because aircraft are not relevant to the PM task when handed off. However, all ongoing ATC tasks could also be slowed by processes such as the rehearsal of PM targets or intended actions (Smith, 2010), or the more subtle general allocation of resources to the PM task and away from the ongoing task set (Smith et al, 2007; Smith, 2008).

We expected to replicate Loft and Remington (2010), and find costs to ongoing tasks when individuals hold PM intentions with no aids compared to control conditions. Furthermore, external aids should only be effective if they cue participants when to allocate preparatory attention. Thus, we did not predict PM error or ongoing task cost to be reduced when participants used static aids compared to no aids. We argue that the Vortac et al. (1995) aids improved PM by increasing the likelihood that participants correctly recalled intended actions (i.e., the alternative destinations to direct PM aircraft), rather than by increasing the likelihood that participants remembered that something needed to be done at the appropriate time. Our focus is on examining whether participants can retrieve intentions at the appropriate time, rather than their ability to recall the content of intentions. For this reason, we made the content of intentions simple (i.e., press the 9 key), and asked participants to recall this content post-test (Loft & Remington, 2010).

In contrast to static aids, flash aids will directly cue participants when to allocate preparatory attention. This should increase the probability that preparatory attention is allocated at the time that deviation from routine is required (Loft & Remington, 2010; Reason, 1990). In addition, there should be less reason for individuals using flash aids to allocate preparatory attention outside the arrival of the PM temporal context that will be cued by the flash aids (Marsh et al. 2006). On this basis we predicted that PM error and costs to ongoing tasks would both be reduced when participants used flash aids compared to no aids.

Method

Participants

Sixty undergraduates, 38 females and 22 males, from the University of Queensland, participated in return for course credit or AU$20, and were tested in groups of one to five. The participants had a mean age of 21.4 years.

ATC-lab^Advanced Task, Materials and Procedure

A screenshot of the ATC-lab^Advanced program (Fothergill, Loft, & Neal, 2009) is depicted in Figure 1. The light polygon area designated the sector and the black lines denoted flight paths. Each aircraft was represented by a symbol (circle) with an attached line indicating the heading of that aircraft. Each aircraft had a data-block attached that specified a call sign, speed, aircraft type, and altitude. At the start of each trial, aircraft appeared on flight paths and then proceeded along routes before crossing sector boundaries and exiting the screen. New aircraft continued to enter throughout the trial. Aircraft positions were updated every second.

Participants accepted and handed-off aircraft, and prevented aircraft conflicts. As aircraft approached within 5nm of the sector they flashed orange to indicate that they needed to be accepted. Participants accepted aircraft by clicking in the aircraft symbol then pressing the A key. Aircraft then turned green. As aircraft left the sector (i.e., crossed the sector boundary), they flashed blue to signal that they needed to be handed-off. Participants handed-off aircraft by clicking in the aircraft symbol then pressing the H key, after which the aircraft turned white. Participants had 20 seconds after an aircraft flashed to accept or hand-off the aircraft.

Participants also detected and prevented conflicts. Conflicts occurred when aircraft simultaneously violated 5 nautical mile lateral and 1000 feet vertical separation. To prevent conflicts, participants changed the altitude of aircraft by left clicking on altitude indicators and choosing a new altitude from a drop down menu. The speed of aircraft could not be changed. If participants failed to prevent a conflict, the aircraft pair turned yellow indicating that the separation standard was violated, and returned to green when separation was re-established.

Participants received instructions for aircraft acceptance and hand-off and conflict prevention, followed by instructions on earning points. Successful aircraft acceptance and hand–off garnered 10 points each, while 10 points were deducted for failures to accept or hand-off aircraft. For successful conflict detection, participants received between 10 and 40 points, depending on speed of responses. Forty points were deducted for conflict misses and unnecessary interventions.

Training

The training phase consisted of eight 5-minute trials presented in a random order, with a 30-second break between trials. On average, 15 aircraft were presented at the start of each trial at varying stages of transition through the sector. Each trial included 15 aircraft to be accepted, eight aircraft to be handed-off, two or three conflicts, and three events in which aircraft traveling at different altitudes violated lateral separation.

Test phase

The test phase included four blocks of two 5-minute trials, with a 30-second break between trials. At the start of each block, participants were presented with one of four test instructions for 40 seconds on the computer screen: control (no PM task), no aid, static aid, or flash aid. All four instruction types informed participants that they were to continue to perform ongoing ATC tasks. In addition, the no aid, static aid, and flash aid instructions informed participants to press the 9 key instead of the A key when accepting aircraft traveling at altitudes greater than 440, aircraft with types less than 400, aircraft with speeds greater than 48, or aircraft with call signs greater than 88. Participants only held one of these intentions per test block.

The no aid instruction also informed participants that they would not be provided with an external aid. The static aid instruction informed participants that they would be given an aid that would contain both the target aircraft and intended action (e.g., if speed greater than 48 then press 9). They were told that this aid would not flash when a target aircraft was flashing for acceptance. The flash aid instruction also informed participants that they would be given an aid that would contain both the target aircraft and intended action. In addition, they were told that this aid would flash orange when a target aircraft (e.g., an aircraft with speed greater than 48) was flashing for acceptance. After each test block that involved a PM component participants were asked to recall the target aircraft and PM response key.

Each test block consisted of two trials each. Each trial contained an air traffic scenario: a certain combination of aircraft, conflicts, and target aircraft. The entry time of non-target aircraft and target aircraft, and the timing and difficulty of conflicts, were held consistent for each trial. Approximately 14 aircraft were presented at the commencement of each test trial at varying stages of transition in the sector. Twenty aircraft were accepted per trial, two or three of which were target aircraft, resulting in a total of five target aircraft and 35 non-target aircraft in each test block. Each trial also included 10 aircraft to be handed-off, three conflicts, and three events in which aircraft traveling at different altitudes violated lateral separation. None of the target aircraft were involved in conflicts. The four different test blocks were presented in the same order for all participants. To control for order effects, the four test instructions (control, no aid, static aid, and flash aid) were presented equally often in each block position².

Results and Discussion

Significance was set at an alpha level of .05. Effect sizes for t tests were estimated using Cohen’s d (Cohen, 1988). Predictions regarding PM and costs to ongoing tasks were tested using planned contrasts that directly paralleled our hypotheses. This approach minimized the potential problem of capitalizing on chance that could arise from statistically parceling the same data in multiple ways (Rosenthal & Rosnow, 1985). In addition, we corrected for family-wise error rate by reporting Bonferroni adjusted p-values (i.e., multiplying each p-value by the number of comparisons made). Based on the findings of Vortac et al. (1995) and Loft and Remington (2010), we had power of 1.0 to detect the large-size effect, and medium-to-large size effect, changes in PM performance and ongoing task performance, respectively (Cohen, 1988).

Scores on each training trial were obtained as a function of the accuracy of aircraft acceptance and hand-off, and the accuracy and speed of conflict detection. Scores obtained on each training trial significantly increased over training, F_linear(1,59) = 28.86, p < .01.

Prospective Memory Performance

During the post-test questionnaire, 98% percent of PM instructions (the target aircraft and intended action) were recalled, with no differences between conditions (ts<1). No errors of omission were made, that is, all PM target aircraft were accepted, either correctly (9 key) or incorrectly (A key). A PM error was defined as the substitution of a routine aircraft acceptance response for a PM instructed response. The PM data are presented in Table 1. PM error was not reduced when participants used a static aid compared to no aid, t<1. However, as predicted, PM errors decreased in the flash condition compared to no aid, t(59) = 3.45, p < .01, d = .62. PM false alarms (i.e., pressing the 9 key to non-target aircraft) were rare (1% of non-target aircraft) and did not differ between conditions, ts<1. In summary, external aids only facilitated PM when the aid flashed at the time deviation from routine was required.

Table 1.

Proportion of Prospective Memory Error, Prospective Memory False Alarm, Conflict Detection Misses, Conflict Detection Time, Aircraft Acceptance Time, and Aircraft Hand-Off Time as a Function of Condition in Experiment 1 and Experiment 2

	PM Error	PM False Alarm	Conflict Miss	Conflict Time	Acceptance Decision Time	Hand-Off Decision Time
Experiment 1
No aid	.14 (.26)	.01 (.03)	.10 (.14)	50.90 (16.07)	.51 (.28)	.41 (.25)
Static Aid	.12 (.24)	.01 (.02)	.12 (.16)	53.31 (17.72)	.48 (.27)	.43 (.27)
Flash Aid	.02 (.08)	.01 (.02)	.11 (.16)	49.38 (16.45)	.48 (.26)	.38 (.25)
Control			.06 (.13)	41.94 (14.65)	.35 (.20)	.37 (.25)
Experiment 2
No aid	.12 (.17)	.02 (.02)	.12 (.12)	56.03 (14.33)	.49 (.23)	.42 (.24)
Static Aid	.14 (.23)	.01 (.01)	.12 (.15)	57.12 (17.49)	.51 (.26)	.43 (.24)
Flash Aids	.02 (.04)	.003 (.006)	.08 (.07)	48.96 (12.18)	.37 (.21)	.44 (.26)
Control			.07 (.09)	48.03 (13.41)	.26 (.17)	.28 (.23)

Note: Standard deviations presented in parentheses.

Ongoing Task Performance

Ongoing task performance was analyzed by examining conflict detection accuracy and response time, aircraft acceptance decision time, and aircraft hand-off decision time. These data are presented in Table 1.

Conflict Detection

There was no significant difference in the number of conflicts missed when comparing the no aid and control conditions, t(59) = 2.10, p = .16, the static aid and no aid conditions, t<1, the flash aid and no aid conditions, t<1, or the flash aid and control conditions, t(59) = 2.34, p = .09. Conflict detection false alarms were made when participants changed the altitude of aircraft not in conflict. Participants made 0.52 false alarms per block, with no effects approaching significance, ts<1. Participants were slower to detect conflicts in the no aid condition compared to the control condition, t(59) = 5.11, p < .01, d = .58. Relative to the condition in which no aid was provided, there was no difference in conflict detection time when participants were provided with a static aid, t(59) = 1.26, p = .85, or flash aid, t<1. Participants were slower to detect conflicts in the flash aid condition compared to the control condition, t(59) = 4.99, p < .01, d = .48.

In summary, there were no significant differences between conditions in conflicts missed, although there were trends for participants to miss more conflicts when they held intentions and were not provided external aids compared to control conditions (4% increase). Neither the static nor the flash aid was effective in reducing the number of conflicts missed. Participants were slower to detect conflicts, relative to the control condition, when they held intentions and were not provided external aids, replicating Loft and Remington (2010). Neither static nor flash aids decreased these costs to conflict detection response time.

Aircraft acceptance decision times

Aircraft acceptance decision times were obtained by calculating the time taken to press the A key after the individual had clicked the symbol of an aircraft that was flashing orange (Loft & Remington, 2010). Our analysis of acceptance decision times excluded target aircraft and non-target aircraft that were flashing for acceptance when target aircraft were approaching the sector boundary. In addition, to avoid response costs associated with post-output PM monitoring processes, we excluded non-target aircraft that were flashing for acceptance within 20 seconds of when a target aircraft had been accepted. For each within-subject condition, we excluded aircraft acceptance times that were greater than 3 SDs from a participant’s grand mean (Loft & Remington, 2010).

Participants were slower to make aircraft acceptance decisions in the no aid condition compared to the control condition, t(59) = 5.13, p < .01, d = .66. Relative to the condition in which no aid was provided, there was no difference in acceptance decision time when participants were provided with a static aid, t(59) = 1.21, p = .92, or flash aid, t(59) = 1.33, p = .75. Participants took significantly more time to make acceptance decisions in the flash condition compared to the control condition, t(59) = 4.98, p < .01, d = .57. In summary, relative to the control condition, participants were slower to make acceptance decisions when they held intentions and were not provided external aids, replicating Loft and Remington (2010). Neither static nor flash aids decreased these costs to acceptance decision time.

Aircraft hand-off decision times

Aircraft hand-off decision times were obtained by calculating the time taken to press the H key after the individual had clicked the aircraft symbol of an aircraft flashing blue. We used the same criteria for excluding aircraft and the same data trimming techniques as for acceptance decision time. There was no difference in hand-off decision time when comparing the no aid and control conditions, t(59) = 1.44, p = .62, the static aid and no aid conditions, t<1, the flash and no aid conditions, t(59) = 1.06, p = 1.17, or the flash and control conditions, t<1. In summary, we found no evidence of a cost to the time taken to make aircraft hand-off decisions when individuals held PM intentions in Experiment 1.

Experiment 2

In Experiment 1 individuals not provided external aids failed to remember to deviate from routine on 14% of occasions and were slower to make acceptance decisions and detect conflicts compared to control conditions. Static aids did not improve performance. PM errors were reduced by flash aids, and we attribute this to the attention grabbing ability of the flashing aid at the time deviation from routine was required. However, costs to ongoing tasks were not reduced by flash aids. There are several possibilities for why the flash aid did not reduce cost. First, it may have been difficult for participants to formulate strategies for using flash aids at the outset of the test blocks after being instructed to expect the presentation of four alternating test conditions (Vortac et al., 1995). In addition, with such limited practice using flash aids (one test block), participants may have found it difficult to ‘trust’ that there were no target aircraft present when the flash aid was silent (Parasuraman & Wickens, 2008). In Experiment 2 we use a between-subjects design to facilitate the use of more focused strategies for using flash aids, and to provide participants with more practice using flash aids. We expected significant reductions in costs to ongoing tasks with the use of flash aids compared to no aids in Experiment 2.

In Experiment 2 we also tested an alternative external aid that we refer to as the ‘coordinated aid’. In Experiment 1, the flash aid may not have reduced ongoing task cost because participants were distracted by the constant (and perhaps unnecessary) presence of the PM instruction on the display. Although the flash aid in Experiment 1 only flashed when the target aircraft were present, the aid remained on the screen during the entire trial and this may have encouraged participants to engage preparatory attention when it was not necessary to do so. The coordinated aid in Experiment 2 was not presented during the entire trial, but rather was automated so that the appearance and flash of the aid was coordinated with the occurrence of a target aircraft flashing for acceptance. In all other respects the coordinated aid was identical to the flash aid, in that it specified both the PM cue and intended action, and it pulsated orange at the time when a target aircraft needed to be accepted.

Method

Participants

Undergraduates from the University of Western Australia participated in return for course credit or AU$20. The 218 participants, 123 females and 95 males, had a mean age of 22.6 years and were tested in groups of one to five. The control, static aid and flash conditions included 44 participants while the no aid and coordinated aid conditions each included 43 participants.

ATC-lab^Advanced Task and Materials and Procedure

The ATC simulation was identical to Experiment 1, with the exception that the test instructions (control, no aid, static aid, flash aid, & coordinated aid) were manipulated between subjects. The coordinated aid instruction informed participants that they would be given an external aid that would appear and flash orange when a target aircraft was flashing at the sector boundary to be accepted. The exact same air traffic scenarios used in Experiment 1 were used in Experiment 2. However, unlike Experiment 1, the presentation order of test blocks was counterbalanced.

Results and Discussion

We had power of .98 and .87 to detect large-size and medium-to-large size effects, respectively, in changes in PM and ongoing task performance (Cohen, 1988). As in Experiment 1, we corrected for family-wise error rate by reporting Bonferroni adjusted p-values.

Due to a technical malfunction the training data was not recorded for three participants in the no aid condition, two participants in the flash aid condition, and one participant in the static aid condition. A 5 condition (control, no aid, static aid, flash aid, coordinated aid) × 8 (training block) ANOVA was conducted on training scores. Training scores obtained on each trial significantly increased over training, F_linear(1,207) = 82.12, p < .01. There was no effect of condition, F(4,207) = 1.24, p = .30, and no interaction between condition and training block, F_linear(4,207) = 1.59, p = .18.

Analysis of PM and ongoing task performance revealed that there were no significant differences between the flash aid condition and the coordinated aid condition, ts<1. This indicates that the effectiveness of flash aids was not influenced by whether the flash aid was present during the entire performance interval or present only when target aircraft were presented. Thus, for brevity, we combined the flash and coordinated aid conditions into a single flash aid condition (aids that flashed when a target aircraft was ready for acceptance) to investigate the effect of flash aids on performance.

Prospective Memory Performance

As in Experiment 1 the use of simple PM intentions resulted in accurate recall of intentions at post-test questioning (98%), with no differences between conditions (ts<1). Also consistent with Experiment 1, no errors of omission were made but there were failures to deviate from acceptance routines. The PM error data are presented in Table 1. Planned contrasts revealed that PM errors were not reduced when participants used static aid compared to no aid, t<1. PM errors decreased when participants used flash aids compared to no aid, t(128) = 5.69, p <.01, d = .87.

Consistent with Experiment 1, participants made false alarms to less than 1% of non-target aircraft accepted. There was no reduction in false alarms when participants used static aids compared to no aid, t(85) = 1.75, p = .25. However, participants made fewer false alarms when using flash aids compared to no aid, t(128) = 5.16, p <.01, d = .83. Fifty-six percent of participants in the no aid condition made at least one false alarm, compared to 24% in flash aid conditions. In summary, consistent with Experiment 1, external aids only reduced PM errors when they were set to flash at the time deviation from routine was required. In addition, individuals made fewer PM false alarms when using flash aids compared to no aids.

Ongoing Task Performance

Ongoing task performance data are presented in Table 1.

Conflict detection

The number of conflicts missed in the no aid condition did not differ from the number missed in the control condition, t(85) = 2.09, p = .20, the static aid condition, t<1, or the flash aid condition, t(128) = 2.14., p = .17. The use of flash aids also did not affect the number of conflicts missed relative to the control condition, t<1. Participants made 0.44 conflict detection false alarms per test block, with no effects approaching significance, ts<1. In short, the number of conflicts missed and the number of conflict false alarms showed no significant evidence of cost. However, as in Experiment 1, there were trends for participants to miss more conflicts when they held intentions and were not provided external aids compared to control conditions (5% increase).

Participants were slower to detect conflicts in the no aid condition compared to the control condition, t(85) = 5.11, p < .01, d = .58, indicating a cost to the ongoing task that was not reduced with the use of a static aid compared to no aid, t<1. However, participants were faster to detect conflicts when using flash aids compared to no aid, t(128) = 2.93, p <.05, d = .53 (Block 1 M_diff = −9.64; Block 4 M_diff = −7.04). Finally, there was no difference in conflict detection time between the flash aid condition and the control condition, t<1 (Block 1 M_diff = 0.77; Block 4 M_diff = 0.58). In summary, replicating Experiment 1, participants were slower to detect conflicts, relative to the control condition, when they held intentions and were not provided external aids, and the use of static aids did not decrease these costs compared to when no aids were used. In contrast, the use of aids that flashed at retrieval reduced costs to conflict detection time.

Aircraft acceptance decision times

Participants were slower to make aircraft acceptance decisions in the no aid condition compared to the control condition, t(85) = 5.47, p < .01, d = 1.17, and this cost was not reduced when using a static aid relative to no aid, t<1. Flash aids did reduce acceptance decision times compared to no aid, t(128) = 2.96, p <.05, d = .54 (Block 1 M_diff = −0.13; Block 4 M_diff = −0.16), but the use of flash aids did not eliminate cost, as response times in the flash aid condition were longer than response times in the control condition, t(129) = 3.10, p <.01, d = .60 (Block 1 M_diff = 0.11; Block 4 M_diff = 0.08). In summary, replicating Experiment 1, participants were slower to accept aircraft, relative to the control condition, when they held intentions and were not provided external aids, and the use of static aids did not decrease these costs compared to when no aids were used. In contrast, the use of aids that flashed at retrieval reduced costs to acceptance decision times.

Aircraft hand-off decision times

Participants were slower to make aircraft hand-off decisions in the no aid condition compared to the control condition, t(85) = 2.78, p <.05, d =.60, and decision times were not reduced by the use of either static or flash aids, as neither condition differed in hand-off decision times from the no aid condition, ts < 1. Further, the flash aid condition had longer hand-off decision times relative to the control condition, t(129) = 3.27, p<.01, d = .65 (Block 1 M_diff = 0.17; Block 4 M_diff = 0.14). In summary, despite the fact that PM responses were not required on hand-off, relative to the control condition, participants were slower to make aircraft hand-off decisions when they held intentions and were not provided external aids. Neither static nor flash aids decreased these costs to hand-off decision time.

General Discussion

We developed and tested the effectiveness of three types of external aids for reducing PM error and cost to ongoing tasks in an ATC simulation. The empirical findings can be summarized as follows. Static aids not uniquely associated with the appropriate opportunity to complete the PM task were not effective in reducing PM error or costs to ongoing tasks. In contrast, aids set to flash when deviation from routine was required reduced PM error (Experiment 1 & 2) and PM false alarms (Experiment 2). In addition, the cost to the ongoing task as measured by both conflict detection time and aircraft acceptance decision time was reduced when participants used aids that flashed at retrieval in Experiment 2. In fact, the numerical difference in response times between the flash aid condition and control condition was not significant in Experiment 2. However, individuals took longer to make aircraft acceptance (Experiment 1 & 2) and hand off (Experiment 2) decisions when using flash aids compared to control conditions. PM and ongoing task performance were not affected by whether flash aids were present during the entire trial or only present when target aircraft were presented. Instead, the crucial determinant of the success of the external aids was whether they were set to flash when deviation from routine was required.

Theoretical Implications

The fact that PM performance was near ceiling (2% error) when external aids flashed at retrieval indicates that these aids were effective in capturing and directing attention. Loft and Remington (2010) argued that participants forget to perform PM tasks because they fail to maintain the preparatory attention required to deviate from aircraft acceptance routine. In our experiments, PM performance improved when individuals were reliably informed by flash aids when target aircraft needed to be accepted. One theoretical interpretation is that flash aids reduced PM error because they ensured that preparatory attention was allocated at the appropriate time during the ongoing task, allowing individuals to engage the recognition memory processes required to detect target aircraft flashing at the sector boundary (Smith et al., 2007). It is also possible that in some instances flash aids lead to the automatic retrieval of intentions (see McDaniel & Einstein, 2007), but our studies are not designed to adjudicate this issue.

Our findings that static aids did not facilitate PM are inconsistent with Vortac et al. (1995). Vortac et al. concluded that having the PM instruction available at retrieval was critical for reducing PM error. However, there were many alternative aircraft destinations in the Vortac et al. task for participants to remember. Thus, participants without aids may have retrieved their intent at the correct time, but forgotten the alternative destination of target aircraft. We have shown that having a PM instruction available at retrieval provides no advantage when the content of intentions is simple. That is, we found no evidence that static aids helped individuals remember at the correct time that they needed to deviate from routine in our ATC simulation.

The cost to ongoing tasks found in the no aid condition provides an important replication of Loft and Remington (2010). Individuals were clearly allocating attentional resources to the PM task at the expense of ongoing tasks. Flash aids allowed participants to devote these resources more selectively, thereby reducing costs. This reduction in costs is consistent with findings in the basic literature that individuals can selectively allocate resources when reliably informed of the temporal context in which PM targets will occur (Marsh et al., 2006). Static aids did not provide this cueing function and thus did not reduce costs to ongoing ATC tasks. The prospect that external aids (reminders) can reduce costs to ongoing tasks has not previously been considered by researchers reporting the ‘benefits’ of reminders (Finstad et al. 2006; Guynn et al., 1998).

As predicted, the between-subjects design used in Experiment 2 allowed participants to use flash aids more effectively than the within-subjects design used in Experiment 1. That is, while flash aids reduced costs to conflict detection time and aircraft acceptance time in Experiment 2, they did not reduce these costs in Experiment 1. The pattern of results in Experiment 2 was present in Block 1 of the test trials, and did not change across blocks. In other words, the benefits of the flash aids appeared early in Experiment 2 and were maintained. Thus, it is unlikely that the lack of reduction in costs in Experiment 1 was due to problems in ‘trusting’ flash aids due to limited experience with the aids, as participants in Experiment 2 showed the benefits in the first block without prior experience (Parasuraman & Wickens, 2008). A more likely explanation is that the use of a between-subject design in Experiment 2 allowed participants to formulate more effective strategies for using flash aids before they started the test trials. In contrast, participants in Experiment 1 were instructed to expect alternating test conditions (and were not instructed what order of presentation to expect), which may have impeded the selection and implementation of effective strategies for utilizing flash aids. It is less clear why we found costs to hand-off decision time between subjects (Experiment 2) but not within subjects (Experiment 1). We have, however, replicated the results of Experiment 2 in several unpublished experiments.

Although the particular static aids used in the current experiment were not beneficial, this does not rule out the possibility that other non-flashing aids that provide additional diagnostic information concerning the target aircraft would be beneficial. In a recent experiment, which was motivated by experienced controllers’ reports that they routinely recognize specific aircraft events that repeatedly occur in the same sector locations (Histon & Hansman, 2002; Loft et al., 2007), Loft, Finnerty, and Remington (under review) found that informing participants of the general sector location were PM aircraft would approach for acceptance reduced PM error rates and reduced costs to ongoing tasks. The location information provide by Loft et al. (under review) allowed individuals to decide where best to allocate preparatory attention. Flash aids in the current study allowed individuals to decide when best to allocate preparatory attention. Static aids used in the current study provided neither type of contextual information. Taken together, the current findings, combined with the recent work by Loft et al. (under review), indicate that the utility of external aids in the ATC task will likely be a function of the extent to which the aids provide information that is diagnostic of target aircraft occurrence.

A critical research question concerns what processes underlie the costs to ongoing ATC tasks. PAM theory proposes that the costs are due, at least in part, to engagement of preparatory attention, which allows the individual to be prepared to make a decision to respond differently to certain stimuli (Smith et al., 2007). However, the exact nature of those processes has yet to be determined and the existence of a cost can be interpreted in different ways (for discussion see Hicks et al., 2005; Loft, Humphreys, & Whitney, 2008). Nevertheless, costs to ongoing tasks have been demonstrated in a variety of tasks (Smith et al., 2007), including the ATC task in Loft and Remington (2010) and the current experiments. Thus, an important question to address is: How can the cost be reduced or eliminated? The current results point to ways to reduce the cost while simultaneously improving PM. This is an encouraging start, but future research should evaluate ways to maintain high levels of PM, accompanied by further reductions of cost to the ongoing ATC tasks.

Practical Implications, Limitations and Conclusions

Operators that continuously monitor multi-item dynamic displays (e.g., ATC, naval radar tracking, air battle management) often perform routine tasks quickly and with little attentional capacity (Klein, 1989; Rasmussen, 1983), making them vulnerable to substituting routine actions for atypical intended actions (Dismukes, 2008; Reason, 1990; Shorrock, 2005). Further, although high levels of PM performance are undoubtedly exhibited by experts in field operations, the costs to ongoing tasks demonstrated here and by Loft and Remington (2010) suggest that such performance may come at a cost to concurrent tasks. As a result, the question of how to develop external aids to decrease PM error and costs to ongoing tasks is a crucial one for practitioners. We made a significant contribution in this regard by demonstrating that the theoretical principles embedded in the experimental psychology literature have utility for designing external aids in an ATC simulation context.

Our data suggest that aids should be designed to command attention at the time that deviation from routine is required. This conclusion is at odds with the fact that controllers typically use static aids such as notepads, ‘post-it’ notes, and free text boxes, to help them remember to perform intentions (Neal & Moiij, 2008). However, while we have successfully applied theoretical principles to design effective external aids for use in an ATC task, our simulations are limited by the use of a student sample and the provision of short training histories. We cannot conclude that our findings with these relatively inexperienced participants generalize to field operations involving experienced controllers. It is likely that in the majority of cases, expert controllers use static aids effectively (Shorrock, 2005) and as noted above, there is evidence that knowledge of the probable location of display information relevant to deferred actions can be used as an effective non-flashing aid (Histon & Hansman, 2002; Loft et al., 2007; Loft et al., under review).

Nonetheless, given the number of aircraft and concomitant actions required from controllers, even small error probabilities with the use of static aids or other non-flashing aids could translate into significant incident rates (Shorrock, 2005). Therefore, any potential benefit from the use of flash aids is worthy of consideration. There undoubtedly exist many practical constraints to consider before implementing flash aids in ATC or other work settings. For example, a flash aid that is salient enough to guarantee attention may be overly distracting and subjectively annoying (Bartram, Ware, & Calvert, 2003), potentially taking an operator’s attention away from other safety-critical tasks. In addition, flashing is already used to attract attention in ATC (e.g., conflict alerts, acceptances), which may reduce the effectiveness of flashing as an attention-getting device. Also if workload is an issue then the extra effort attaching a ‘time tag’ to an external aid might be too costly. This latter issue could potentially be overcome by developing flash aids that do not require the controller to predict the time at which actions should be taken, but instead require the more general prediction of the location (or relative aircraft positions) at which actions should be taken.

Another limitation is that our PM task was established by the experimenter. Not always, but often, PM tasks in ATC are negotiated between controllers and pilots on a moment-to-moment basis. Thus, future research needs to examine whether aids are as effective for reducing PM error and ongoing task cost when PM tasks are encoded ‘on the fly’ by participants. Finally, we presented targets more frequently than would be the case in operational settings. Much of the difficulty in remembering to perform intentions may be related to the infrequency in which targets are presented (Loft & Yeo, 2007), and future research should examine whether the effectiveness of external aids interacts with target frequency.

Many safety critical work systems in the aviation, process control, and medical domains require operators to act in parallel with automation (Durso & Sethumadhavan, 2008; Parasuraman & Wickens, 2008). If highly-reliable automation can be designed to take responsibility for a given task, then the human operator can reallocate attention, allowing for more successful multitasking (Dixon, Wickens, & McCarley, 2007). Our findings are consistent with this notion. However, the extent to which automated aids improve performance in operational settings is difficult to predict given that aids are unlikely to be 100% reliable (Metzger & Parasurman, 2005). Miss-prone automation degrades reliance, and as a result can lead to decrements in concurrent task performance because operators need to allocate attention to monitoring the raw data behind the automation (Dixon & Wickens, 2006). In addition, highly reliable low-miss automation, although availing ample resources for concurrent tasks, may leave the operator quite vulnerable to the rare automation misses (Moray & Inagaki, 2000). In ATC for example, a loss of situational awareness is a common problem when automation such as conflict detection aids or clearance advisory aids fail to operate (Durso & Manning, 2009). It is imperative then that future research examines the effects that different levels of reliability of an external aid have on an individual’s trust in and behavioral dependence on that aid, and subsequent effects on PM error and ongoing task performance.

In conclusion, the design of external automated aids for supporting performance in complex dynamic work systems is a diverse and challenging problem. It is essential that designs of these automation tools are based on a thorough analysis of human cognition and decision-making processes. We make a significant contribution in this regard by successfully applying theories and methods from the experimental literature to demonstrate the potential relative utility of external aids for occupations where individuals monitor multi-item dynamic displays.