Minimizing the Disruptive Effects of Prospective Memory in Simulated Air Traffic Control

Abstract

Prospective memory refers to remembering to perform an intended action in the future. Failures of prospective memory can occur in air traffic control. In two experiments, we examined the utility of external aids for facilitating air traffic management in a simulated air traffic control task with prospective memory requirements. Participants accepted and handed-off aircraft and detected aircraft conflicts. The prospective memory task involved remembering to deviate from a routine operating procedure when accepting target aircraft. External aids that contained details of the prospective memory task appeared and flashed when target aircraft needed acceptance. In Experiment 1, external aids presented either adjacent or non-adjacent to each of the 20 target aircraft presented over the 40min test phase reduced prospective memory error by 11% compared to a condition without external aids. In Experiment 2, only a single target aircraft was presented a significant time (39min–42min) after presentation of the prospective memory instruction, and the external aids reduced prospective memory error by 34%. In both experiments, costs to the efficiency of non-prospective memory air traffic management (non-target aircraft acceptance response time, conflict detection response time) were reduced by non-adjacent aids compared to no aids or adjacent aids. In contrast, in both experiments, the efficiency of the prospective memory air traffic management (target aircraft acceptance response time) was facilitated by adjacent aids compared to non-adjacent aids. Together, these findings have potential implications for the design of automated alerting systems to maximize multi-task performance in work settings where operators monitor and control demanding perceptual displays.

Prospective memory (PM) refers to the task of remembering to perform planned actions at appropriate points in the future. In many work contexts, such as in aviation and medical practice, the failure of operators to remember to complete deferred actions can have serious consequences (Dismukes, 2012). In air traffic control (ATC) for example, PM failures have contributed to the loss of safe separation between aircraft (Shorrock, 2005). A type of PM error that frequently appears in anecdotes and incident reports is the failure of a controller to substitute an atypical intended action for a routine action. For example, a controller may intend to deviate from normal procedure and hold an aircraft at a specific future waypoint (due to crossing traffic), but when that aircraft reaches that waypoint may forget the intention and instead instruct the pilot to routinely descend. In addition to the potential for PM error, the cognitive resources needed to maintain and retrieve deferred actions can decrease the efficiency with which controllers perform concurrent non-PM air traffic management tasks (Dismukes, 2011; Loft, Sanderson, Neal, & Mooij, 2007). This decrease in the efficiency of non-PM air traffic management task performance is referred to in the current paper as the ‘cost to non-PM air traffic management’.

Only two previous experimental studies have examined whether external memory aids can reduce PM error and also alleviate costs to non-PM air traffic management (Loft, Smith & Bhaskara, 2011; Vortac, Edwards, & Manning, 1995). In these studies external aids reduced PM error, but did not eliminate costs to non-PM air traffic management. In the current paper we extend this previous research in several important ways. First, we examine whether the location in which external aids appear on the ATC display influences the magnitude of cost to non-PM air traffic management. To do so, we introduce new measures of non-PM air traffic management that take into account visual scanning and anticipation strategies. Second, we examine the impact of the location of external aids on the efficiency, not just success, of PM task performance (PM air traffic management) by measuring the time to make successful PM responses. Third, we examine the effectiveness of external aids when only a single PM target aircraft event was presented a significant time after encoding of the PM instruction.

External Aids in Simulated Air Traffic Control

We have recently applied theories and methods from the basic PM literature to simulations of ATC (Loft, Finnerty, & Remington, 2011; Loft, Pearcey, & Remington, 2011; Loft & Remington, 2010; Loft, Smith et al., 2011). In these simulations (see Figure 1), participants are required to accept aircraft entering the sector, to detect and resolve aircraft conflicts (aircraft are in conflict if they will simultaneously violate lateral and vertical separation in the future), and to hand-off aircraft exiting the sector. The PM task consists of remembering to deviate from a routine operating procedure (by pressing an alternative response key) when accepting “target” aircraft that display certain flight data (e.g., altitude > 44,000ft). In addition to examining PM errors, which occur when the routine response is made instead of the alternative PM response for target aircraft, we have examined costs to non-PM air traffic management. Our prior studies indicate that PM task requirements produce a robust cost to non-PM air traffic management, typically in the form of slowed non-target aircraft acceptance, slowed non-target aircraft hand-off, and slowed conflict detection (Loft, Finnerty, et al., 2011; Loft, Pearcy, et al, 2011; Loft & Remington, 2010; Loft, Smith et al., 2011).

Figure 1.

A screenshot of the ATC-labAdvanced program with a non-adjacent external aid. Inbound aircraft were black (e.g., aircraft C66) as they approached the sector, flashing orange for acceptance (C64) when they reached within 5 miles of the sector boundary. Aircraft turned green (C41) when accepted. When outbound aircraft crossed the sector boundary they flashed blue (C26), and then turned white (C30) when handed off. Aircraft were in conflict if they were travelling at the same altitude and would violate the minimum lateral separation of 5nm in the future. The example in Figure 1 shows that the individual had changed the altitude of C55 from 340 to 300 in order to avoid a conflict with C37. In the top right corner, C62 is in conflict with C93 (i.e., they will violate separation in the future if not intervened). In this example the non-adjacent external aid has appeared and is flashing to remind participants they need to press the 9 key for aircraft with speed greater than 48 (aircraft C64 has a speed greater than 48 and is currently flashing for acceptance). The running score (115 points) is presented in the middle right hand side of the display.

Loft, Smith et al. (2011) provided participants with external aids to reduce PM error and reduce costs to non-PM air traffic management. These external aids, which presented the details of target aircraft and the intended action (e.g., press 9 if speed > 48), were presented in the middle of the right hand side of the display and either flashed, or remained static, when target aircraft flashed for acceptance. Static aids did not reduce PM error or costs to non-PM air traffic management compared to when no aids were provided. Crucially, the static aid did not provide a salient sensory property that signalled the presence of target aircraft¹. In contrast, the external aids that flashed when target aircraft flashed for acceptance (1) reduced PM error, (2) reduced costs to conflict detection and non-target aircraft acceptance, but (3) did not reduce costs to non-target aircraft hand-off. Loft, Smith et al. concluded that the flash aids were effective because they alerted participants when they needed to deviate from routine (also see Byrne, 2008; Chung & Byrne, 2008; external aids for postcompletion errors). However, because significant costs to non-PM air traffic management tasks remained, there is potential for improvement in the design of flash aids (Experiment 1 and 2). It is also crucial to examine the impact of flash aids on the efficiency of PM task performance (Experiment 1 and 2), and to test the utility of flash aids under additional task conditions (Experiment 2).

Will the Location of External Aids Affect Non-PM Air Traffic Management?

Australian air traffic controllers interviewed by the first author claimed that external aids would be more effective if presented adjacent to the data block (flight information) of target aircraft, thereby drawing attention directly to the PM relevant display region. They further suggested that laboratory experimentation manipulating the location of external aids was crucial for increasing the operational relevance of our research program. Experiment 1 compared external aids presented adjacent to target aircraft (see Figure 2) with the flash aids used in the Loft, Smith et al. (2011) study, which were presented on the side of the ATC display (see Figure 1). We refer to these as adjacent and non-adjacent aids, respectively. Loft et al. (2011) found that external aids were equally effective irrespective of whether they were constantly presented on the display and then flashed when a target aircraft flashed for acceptance, or whether they only appeared on the display (and flashed) when a target aircraft flashed for acceptance. In the current experiments we used the latter method of presenting external aids in order to minimize display clutter (Steelman, McCarley, & Wickens, 2011). Thus, the external aids appeared on the display (either adjacent or non-adjacent to target aircraft), and flashed orange, when a target aircraft flashed for acceptance at the sector boundary. As outlined below, we anticipated that the use of adjacent aids would decrease costs to non-PM air traffic management compared to the use of non-adjacent aids.

Figure 2.

A screenshot of the ATC-labAdvanced program with an adjacent external aid. The adjacent external aid has appeared and is flashing adjacent to C64 to remind participants they need to press the 9 key for aircraft with speed greater than 48 (aircraft C64 has a speed greater than 48 and is currently flashing for acceptance).

Costs to non-PM air traffic management in the Loft, Smith et al. (2011) study indicate that participants were allocating cognitive resources to the PM task (Burgess & Shallice, 1997; McDaniel & Einstein, 2007; Savine, McDaniel, Shelton, & Scullin, 2012; Smith, 2008) prior to the onset of non-adjacent aids on the ATC display. These cognitive resources might be required to detect the non-adjacent external aid. It has been shown that salient perceptual events do not reliably capture attention unless their display properties match the ‘attentional set’ that participants are using to satisfy more routine (immediate) behavioural goals (contingent orienting; Al-Aidroos, Harrison, & Pratt, 2010; Folk, Remington & Johnston; 1992; Folk, Remington & Wright, 1994; Pashler, Johnston, & Ruthruff, 2001). To perform the routine tasks of aircraft acceptance, aircraft hand-off and conflict detection, participants likely employ an attentional set that prioritizes the processing of aircraft approaching or flashing at the sector boundary (see Loft & Remington, 2010; Rantanen & Nunes, 2005). Non-adjacent aids are presented in the periphery of this attentional set, and cognitive resources might thus be required to detect the non-adjacent external aid. In contrast, adjacent aids are essentially presented as part of the same object as aircraft flashing for acceptance at the sector boundary, and thus the processing of adjacent aids is more central to the attentional set employed to perform routine ATC tasks. Adjacent aids therefore may more reliably capture attention. In summary then, we expected that the provision of adjacent aids, compared to non-adjacent aids, would reduce the need for participants to devote cognitive resources to the PM task prior to external aid onset, further reducing costs to non-PM air traffic management. We also expected to replicate the reduction in cost to non-PM air traffic management reported by Loft, Smith et al. (2011) when participants use non-adjacent aids compared to no aids.

To test these predictions, we improved the manner in which we measured non-PM air traffic management. Loft, Smith et al. (2011) calculated non-target aircraft acceptance time and non-target hand-off aircraft time as the time taken to press the A key (for acceptance) or H key (for hand-off) after clicking (selecting) the symbol of a non-target aircraft flashing for acceptance or hand-off. However, external aids may also allow participants to more efficiently process the flight data of aircraft approaching the sector boundary before formally selecting aircraft for acceptance. Such anticipation strategies are crucial in operational settings because they allow controllers to ‘stay ahead’ of incoming aircraft and make proactive decisions (e.g., select appropriate routings, apply sector-specific rules, time pilot instructions) and to better cope with future problems (e.g., adverse weather conditions, dense traffic at the destination airport, cross winds at landings; Durso & Manning, 2009; Kontogiannis & Malakis, 2009; Loft et al., 2007). In order to take proactive anticipation strategies into account, we measured non-target acceptance/hand-off time as the total time taken to accept/hand-off aircraft after they first flashed for acceptance/hand-off. The other measures of non-PM air traffic management were the accuracy and speed of aircraft conflict detection.

Will the Location of External Aids Affect PM Air Traffic Management?

We did not anticipate a difference in PM error between the adjacent and non-adjacent aid conditions given that Loft, Smith et al. reported a PM error rate of only 1–2% with non-adjacent aids. If PM errors are again near floor with the provision of external aids, we would not expect any difference in PM error between the adjacent and non-adjacent aid conditions.

We introduce a measure of the efficiency of PM air traffic management by calculating the time taken to accept target aircraft with the PM response after they had flashed for acceptance (referred to hereafter as ‘target acceptance time’). Participants need to locate the flashing target aircraft, verify the target status of that aircraft, and retrieve and execute the PM action (“9” key). The PM action was identical for all PM conditions. Therefore, any differences in target acceptance time between conditions should reflect the resource requirements associated with locating and verifying the target status of aircraft (see Loft & Yeo, 2007). Target aircraft acceptance should be faster with the provision of external aids compared to no aids because external aids provide temporal context for the PM task. In other words, when an external aid flashes it indicates that a target aircraft is also flashing for acceptance so a decision about whether a target aircraft is present is not required. Participants should accept target aircraft faster when using adjacent aids compared to non-adjacent aids because in addition to providing temporal context for the PM task adjacent aids are diagnostic regarding the location of target aircraft (also see Loft, Finnerty et al., 2011). Specifically, the adjacent aid draws attention directly to the target aircraft, increasing the likelihood that external aid and target aircraft can be processed as a single object (Awh, Dhaliwal, Christensen, & Matsukura, 2001; Duncan, 1984). In contrast, non-adjacent aids draw attention to the periphery of ATC display, requiring participants to re-orientate their attention to find the target aircraft flashing at the sector boundary.

Experiment 1

Method

Participants

Undergraduates (N= 188; 112 female; mean age = 20.7 years) from the University of Western Australia participated in return for course credit. There were 48 participants in each of the no aid and adjacent aid conditions, and 46 participants in each of the non-adjacent aid and control conditions.

ATC-lab^Advanced task, materials and procedure

The ATC-lab^Advanced task (Fothergill, Loft, & Neal, 2009), material and procedure were identical to Experiment 2 of Loft, Smith et al. (2011), with the exception of the adjacent aid condition. As illustrated in Figures 1 and 2, the sector was shown as a light polygon area with lines that denoted flight paths. Each aircraft was represented by a circle and data block containing flight data such as call sign, aircraft type, speed and both current and cleared altitude. The position of aircraft on flight paths was updated once per second, and the direction of aircraft flight could be ascertained by the aircraft leader line. The ATC simulator recorded response times by rounding down to the nearest second.

Aircraft were black when they first appeared on the display, then flashed orange when they were within 5 miles of the sector boundary and needed acceptance. Participants accepted the aircraft by clicking on the aircraft symbol and pressing the A key. Aircraft then were presented in steady green. As aircraft exited the sector (crossed the sector boundary) they flashed blue and participants selected them and pressed H. Participants were given 10 points if they accepted or handed-off an aircraft within 20 seconds of the aircraft flashing. If they failed to do so they lost 10 points and the aircraft was automatically accepted or handed-off by the ATC simulation.

Aircraft pairs were in conflict if they would simultaneously violate lateral and vertical separation in the future. Participants resolved conflicts by changing aircraft altitude (aircraft speed was not under participant control). If participants failed to detect and resolve a conflict, the aircraft turned yellow when they violated separation and 40 points were deducted. Aircraft returned to a steady green state when minimum separation between the aircraft was re-established. For successfully detected conflicts, participants received between 10 and 40 points, depending on the speed at which the conflict was detected. Forty points were deducted for unnecessary altitude interventions. To increase motivation, participants were told that their final ATC scores would be compared to other participants at the end of the testing session in a de-identified manner. At the end of the session we announced the highest to lowest scores (without naming the individuals who obtained these scores).

Training

Eight 5-minute training trials were presented in a random order, with a 30-second break between trials. On average, 15 aircraft were presented at the start of each trial at varying stages of sector transition. Fifteen aircraft flashed for acceptance, eight aircraft flashed for hand-off, two or three pairs of aircraft were scripted to conflict, and three events presented were aircraft travelling at different altitudes violating lateral separation.

Test phase

Four blocks of test trials were presented, with each block containing two 5-minute trials and a 30-second break between trials. At the start of each test block, participants were presented one of four test instructions for 40 seconds on the computer screen: control (no PM task), no aid, non-adjacent aid, or adjacent aid. Participants in all conditions were instructed to continue to accept aircraft, hand-off aircraft and detect conflicts. The PM conditions were also told that they were required to remember to press the 9 key instead of the A key when accepting aircraft with types < 400, call signs > 88, altitudes > 440, or speeds > 48. Participants in PM conditions only held one of these intentions per test block (overall each participant was presented each PM instruction type once over the four test blocks). The assignment of PM task instruction type (i.e., type, call sign, altitude, speed) across the order of the four test blocks was counterbalanced across the participants.

The non-adjacent aid instruction further informed participants that an external aid would appear on the right hand side of the display (and would contain both the target aircraft and intended action details) and flash orange when a target aircraft was flashing at the sector boundary to be accepted. The adjacent aid instruction was identical, except that participants were informed that the external aid would be presented directly adjacent to the target aircraft flashing at the sector boundary. Participants in the respective conditions were shown an example of their external aids in an ATC simulation demonstration and were told that the external aids would be 100% reliable. It was further reiterated to participants that this meant the external aids would provide all the information required and that they in fact did not have to explicitly remember the PM instruction or search for target aircraft.

Approximately 14 aircraft were present on the display at the commencement of each test trial. Twenty aircraft were accepted per trial, two or three of which were target aircraft, resulting in a total of five target aircraft and 35 non-target aircraft in each test block. Each trial also included 10 aircraft to be handed-off, three conflicts, and three events were presented in which aircraft travelling at different altitudes violated lateral separation. Target aircraft were not involved in conflicts. After each test block participants in PM conditions were asked to recall the target aircraft and PM response key. The presentation order of test blocks was counterbalanced.

Results and Discussion

As described in the method section above, participants earned points (e.g., 10 to 40 points for each successfully detected conflict) or lost points (e.g., minus 40 points for a missed conflict) depending upon their performance on the ATC tasks. The points earned and lost were summed to create a score for each of the eight training trials. A 4 (condition: control, no aid, non-adjacent aid, adjacent aid) × 8 (training trial) ANOVA was conducted on the training scores. Scores obtained on each training trial increased with practice (Trial 1, M = 186.66, SD = 126.74; Trial 8, M = 285.26, SD = 53.57), F_linear(1,184) = 128.87, p < .001, η_p² = .41. There was no effect of condition, F(3,184) = 1.89, p = .13, and no condition by training trial interaction, F_linear(3, 184) = 1.84, p = .14.

For analysis of the PM air traffic management and non-PM air traffic management data from the test phase of the experiment, we used planned contrasts that directly paralleled our hypotheses (Rosenthal & Rosnow, 1985) and corrected for family-wise error rate by reporting Bonferroni adjusted p-values (i.e., multiplying each p-value by the number of comparisons made). Based on the findings in Experiment 2 of Loft, Smith et al. (2011), we had power of .99 to detect large-size effects of external aids on PM error and power of .93 to detect medium-to-large size effects of external aids on non-PM air traffic management (Cohen, 1988). Cohen’s d was used to estimate effect sizes.

PM air traffic management

PM air traffic management was analysed by examining PM errors and target acceptance time. These data are presented in Table 1.

Table 1.

Prospective Memory Error, Target Acceptance Time, Conflict Detection Misses, Conflict Detection Time, Non-Target Acceptance Time, and Non-Target Hand-Off Time as a Function of Condition in Experiment 1 and Experiment 2.

	PM Error	Target Acceptance Time	Conflict Detection Miss	Conflict Detection Time	Non-Target Acceptance Time	Non-Target Hand-Off Time
Experiment 1
No aid	.12 (.15)	3.09 (.92)	.10 (.09)	52.07 (13.25)	2.97 (.92)	2.64 (.87)
Non-adjacent aid	.02 (.06)	2.67 (.58)	.06 (.08)	44.52 (13.78)	2.56 (.68)	2.52 (.66)
Adjacent Aid	.01 (.02)	2.40 (.51)	.08 (.08)	51.37 (11.37)	2.94 (.77)	2.54 (.74)
Control			.06 (.05)	44.44 (13.44)	2.49 (.57)	2.24 (.53)
Experiment 2
No aid	.38 (.49)	3.16 (2.17)	.08 (.09)	46.03 (11.77)	2.73 (.87)	2.50 (.76)
Non-adjacent aid	.04 (.20)	4.56 (2.32)	.08 (.10)	45.55 (13.15)	2.33 (.57)	2.57 (.84)
Adjacent Aid	.04 (.20)	3.61 (1.45)	.07 (.07)	46.43 (13.36)	2.68 (.70)	2.52 (.68)
Control			.06 (.05)	44.77 (13.72)	2.22 (.55)	2.11 (.65)

Notes: The units of measurement for PM error and Conflict Detection Misses are proportion of errors/misses out of total number of opportunities for error/misses over test. The units of measurement for Target Acceptance Time, Conflict Detection Time, Non-Target Acceptance Time and Non-Target Hand-Off Time are seconds. Standard deviations presented in parentheses.

PM errors

During the post-test questionnaire, 97% of PM instructions (the target aircraft features and intended action) were correctly recalled, with no differences between conditions, ts<1. All target aircraft were accepted, either correctly (9 key) or incorrectly (A key). A PM error was defined as the substitution of a routine aircraft acceptance response for a PM response. PM errors decreased in the external aid conditions compared to the no aid condition, t(140) = 6.31, p < .001, d = .96 (Mdiff = 11%). PM errors did not differ between the adjacent aid and non-adjacent aid conditions, t(92) = 1.79, p = .15. Participants made PM false alarms (i.e., pressing the 9 key to non-target aircraft) to less than 1% of non-target aircraft. Participants made fewer false alarms when using external aids compared to no aids, t(140) = 5.80, p < .001, d = .87. Seventy-one percent of participants in the no aid condition made at least one false alarm, compared to 21% in the external aid conditions. There was no difference between the adjacent aid and non-adjacent aid conditions in PM false alarms, t(92) = 1.22, p = .91. In summary, replicating Loft, Smith et al. (2011), external aids that flashed when deviation from routine was required reduced PM error and PM false alarms compared to when no aids were provided. Whether these aids were adjacent or non-adjacent to target aircraft had no effect, in both cases PM error rates were at floor.

PM target acceptance time

Target acceptance time was defined as the time taken to select the target aircraft and press the 9 key after the target aircraft flashed for acceptance. Only target aircraft to which participants made the correct response (“9” key) were included. Target acceptance times that were greater than 3 SDs from a participant’s grand mean were excluded (1.4% of target acceptance times). Participants in the external aid conditions were faster to accept target aircraft than participants in the no aid condition, t(140) = 4.50, p < .001, d = .73, Mdiff = 0.56sec. Participants in the adjacent aid condition were faster to accept target aircraft than participants in the non-adjacent aid condition, t(92) = 2.32, p = .046, d = .49, Mdiff = 0.26sec.

Non-PM air traffic management

Non-PM air traffic management was analysed by examining conflict detection accuracy and response time, non-target acceptance time, and non-target hand-off time. These data are presented in Table 1.

Conflict detection

Conflict detection false alarms were made when participants changed the altitude of aircraft not in conflict. Participants made an average 1.84 conflict detection false alarms across all test blocks, with no effects approaching significance (all ps > .84).

Participants in the no aid condition missed significantly more conflicts, t(92) = 2.92, p = .02, d = .61, Mdiff = 4%, and were slower to detect conflicts, t(92) = 2.77, p = .02, d = .57, Mdiff = 8.26 secs, than participants in the control condition, replicating previous studies (e.g., Loft, Finnerty et al., 2011; Loft, Smith et al., 2011). There was a non-significant trend towards an increased number of conflicts missed for the no aid condition compared to the non-adjacent aid condition, t(92) = 2.34, p = .06. Replicating Loft, Smith et al. (2011), participants in the no aid condition were slower to detect conflicts than participants in the non-adjacent aid condition, t(92) = 2.71, p = .02, d = .56, Mdiff = 7.55 secs. Participants in the adjacent and non-adjacent conditions did not differ in conflict detection misses, t(92) = 1.14, p = .77. However, participants in the adjacent aid condition were slower to detect conflicts than participants in the non-adjacent aid condition, t(92) = 2.63, p = .03, d = .54, Mdiff = 6.85 secs.

Non-target acceptance time

Non-target acceptance time was defined by the time taken to select non-target aircraft and press the A key after the aircraft flashed for acceptance. Non-target acceptance times that were greater than 3 SDs from a participant’s grand mean were excluded (2.3% of non-target acceptance times)². Participants in the no aid condition were slower to accept non-target aircraft than participants in the control condition, t(92) = 3.01, p =.01, d = .63, Mdiff = 0.48sec, replicating previous studies (e.g., Loft, Finnerty et al., 2011; Loft, Smith et al., 2011). Replicating Loft, Smith et al. (2011), participants in the non-adjacent aid condition were faster to accept non-target aircraft relative to participants in the no aid condition, t(92) = 2.47, p = .045, d = 0.51, Mdiff = 0.41sec. Participants in the non-adjacent aid condition were faster to accept non-target aircraft than participants in the adjacent aid condition, t(92) = 2.59, p = .04, d = .53, Mdiff = 0.39sec.

Non-target hand-off times

Non-target aircraft hand-off time was defined as the time taken to select the non-target aircraft and press the H key after the aircraft flashed for hand-off. Hand-off times that were greater than 3 SDs from a participant’s grand mean were excluded (2.1% non-target hand-off times). Participants in the no aid condition were slower to hand-off aircraft than participants in the control condition, t(92) = 2.75, p = .02, d = .56, Mdiff = 0.40 secs. Consistent with Loft, Smith et al. (2011), there was no difference in non-target aircraft hand-off time between the non-adjacent aid and no aid conditions, t<1. There was also no difference between the adjacent and non-adjacent conditions in non-target aircraft hand-off time, t<1.

Summary of Experiment 1

External aids decreased PM errors and PM false alarms relative to when no aids were provided, but it made no difference whether the aids were adjacent or non-adjacent to target aircraft. However, adjacent aids increased the efficiency of PM air traffic management by increasing the speed of target aircraft acceptance compared to non-adjacent aids and no aids. There were costs to non-PM air traffic management tasks. Specifically, non-target acceptance, non-target hand-off, and conflict detection times were slower when participants had PM task requirements and were not provided external aids, compared to participants in the control condition without PM task requirements. The no aid condition also missed more conflicts relative to the control condition. The use of non-adjacent aids decreased the costs to non-PM air traffic management, at least with respect to non-target aircraft acceptance and conflict detection times, compared to the use of adjacent aids or no aids. This finding contradicts our prediction that adjacent aids would reduce the cost to non-PM air traffic management compared to the use of non-adjacent aids or no aids, a point we return to in the General Discussion. Costs to aircraft hand-off were not reduced by the use of external aids.

Experiment 1 makes two significant contributions. First, we replicated the entire set of significant effects reported by Loft, Smith et al. (2011) relating to reduced PM error and reduced costs to non-PM air traffic management with the use of non-adjacent aids compared to no aids. The scientific value of exact replication in establishing the reliability of effect has received much recent attention in the psychological literature (Pashler & Wagenmakers, 2012; Yong, 2012). Replication is particularly vital when the resulting knowledge will potentially be used by human factors practitioners in safety-critical work settings (Jones, Derby, & Schmidlin, 2010). Second, we combined this direct replication with a theoretically and practically relevant extension in the form of the adjacent aid condition. Importantly, we demonstrated that although using adjacent aids, compared to using non-adjacent aids, facilitated PM air traffic management in the form of faster target aircraft acceptance, these adjacent aids also produced a greater cost to non-PM air traffic management relative to non-adjacent aids. Thus, we demonstrated that the placement of external aids had a differential impact on PM air traffic management versus non-PM air traffic management.

Experiment 2

In Experiment 1, new PM instructions were presented every two trials, informing participants of the new flight details for target aircraft that would be presented over the next two trials (i.e, over the next 10 minutes). According to controllers, this rate of PM instruction presentation emulates many ATC settings where successive task demands, requests from pilots, or requests from fellow controllers are encountered. Participants were then presented with five target events during the two trials (10 minutes) that matched the preceding PM instruction. Controllers concurred that this level of target frequency reflects many non-laboratory situations, for example controllers may hold the intention to assign all incoming 767 aircraft from the south of the sector to an atypical altitude. Despite the applicability of the methods used in Experiment 1 to many non-laboratory PM tasks, the controllers suggested that in some operational contexts the first, and often only, PM target event matching a particular delayed intention instruction is encountered after a delay of many minutes or even hours (also see Dismukes, 2012; Shorrock, 2005). Controllers claimed it was these situations where they are most vulnerable to PM error. Motivated by this applied concern, in Experiment 2 we presented a single PM target a significant time (39–42 min) after the presentation of a single PM instruction.

Studies in the basic PM literature have shown that costs to non-PM (ongoing) tasks are reduced with increased ongoing task duration prior to either the refreshment of PM instructions or the presentation of target events (Martin, Brown, & Hicks, 2011; Loft, Kearney, & Remington, 2008; Scullin, McDaniel, Shelton, & Lee, 2010), indicating a reduction in the allocation of resources to the PM task. Thus, a key question in Experiment 2 concerned the extent to which costs to non-PM air traffic management would be observed for the no aid condition relative to the control condition. If costs to non-PM air traffic management continue to be observed in the no aid condition, we expect to replicate the reduction in cost to non-PM air traffic management when participants use non-adjacent aids, compared to when participants use no aids or adjacent aids.

If costs to non-PM air traffic management are reduced in Experiment 2 relative to the levels seen in the no aid conditions of Experiment 1 and prior studies, this would indicate that fewer resources are allocated to the PM task, and we would in turn expect to find quite large PM error rates when no aids are provided. For example, Scullin et al. (2010) reported very high PM error rates (82%) when the first PM target was presented after 500 trials of an ongoing lexical decision task. The poor PM performance reported by Scullin et al. is also consistent with associative activation theories of human cognition which assume that task goals are stored in long-term memory in associative networks and that the baseline activation of task goals decreases with increased delay between task goal encoding and task goal retrieval (Altman & Trafton, 2002; Anderson & Lebiere, 1998; Nowinski & Dismukes, 2005). Thus, a second key question in Experiment 2 concerned whether external aids could compensate for a reduction in resources allocated to the PM task, or a reduction in PM task goal activation, and reduce PM errors by cuing attention to a single target aircraft presented a significant time after the presentation of a PM instruction. Finally, we expected to replicate the finding from Experiment 1 of faster target acceptance times with the provision of adjacent aids compared to non-adjacent aids.

Method

Participants

Undergraduates (N = 190; 124 female; mean age = 20.4 years) from the University of Western Australia participated in return for course credit. There were 48 participants in each of the no aid and adjacent aid conditions, and 47 participants in each of the non-adjacent aid and control conditions.

ATC-lab^Advanced task and materials and procedure

The method was the same as in Experiment 1 but with the following exceptions. Participants in the PM condition were only presented a PM instructions on one occasion, before the commencement of first test block. This instruction asked participants to remember to press the 9 key instead of the A key when accepting aircraft with speeds < 37, call signs > 95, altitudes < 290, or speeds > 53 (participants only held one of these intentions for the entire test phase). The same air traffic scenarios, including the details of conflicts, used in Experiment 1 were used in Experiment 2, except that only one target aircraft was presented during the test phase, during the last trial of the last test block. Reducing the number of targets from 20 to one required some flight details (call signs, altitudes speeds, types) of the 19 target aircraft presented in Experiment 1 to be altered for Experiment 2 so that these aircraft no longer met the PM target criteria. After the test phase had finished participants in PM conditions were asked (a) whether they recalled having to remember to complete an alternative action when accepting certain target aircraft, and if “yes” (b) to recall the target aircraft and PM response key.

Results and Discussion

Scores on each of the eight training trials were calculated using the same method as Experiment 1. A 4 (condition) × 8 (training trial) ANOVA was conducted on the training trial scores. Scores obtained on each training trial increased with practice (Trial 1, M = 208.58, SD = 101.86, Trial 8, M = 287.77, SD = 47.88), F_linear(3,186) = 199.44, p < .001, η_p² = .52. There was no effect of condition, F<1, and no condition by block interaction, F_linear(3, 186) = 1.45, p = .23.

We had the same power as Experiment 1 to detect changes in costs to the efficiency of non-PM air traffic management. We conducted planned contrasts with Bonferroni adjusted p-values to analyse the non-PM air traffic management data. Because we only had one observation per participant for PM performance, PM errors were analysed using χ² and we applied Bonferroni adjusted p-values. Effect sizes for χ² tests are given by phi (ϕ).

PM air traffic management

PM air traffic management was analysed by examining PM error and target acceptance time. These data are presented in Table 1.

PM error

As in Experiment 1, all target aircraft were accepted either correctly (press 9 key) or incorrectly (press A key). At the end of the test phase, all participants in the PM conditions correctly recalled the need to respond differently when accepting target aircraft. Ninety-three percent of participants also correctly recalled the content of the PM instruction, with no differences between conditions (ps > .49 for all χ² tests). The PM data are presented in Table 1. χ² tests revealed significantly fewer PM errors in the external aid conditions compared to the no aid condition, χ² (1) = 27.15, p < .001, ϕ² = .44. In the no aid condition, 18 out of 48 participants (38%) failed to make the PM response, compared to four of 95 participants in the external aid conditions (4%). Two participants failed to make the PM response in each of the external aid conditions. Participants made PM false alarms to less than 0.22% of non-target aircraft accepted. PM false alarms were less frequent when participants used external aids compared to no aids, t(141) = 2.42, p = .03, d = .87. Twenty-nine percent of participants in the no aid condition made at least one false alarm, compared to 18% in the external aid conditions. There was no difference between the adjacent aid and non-adjacent aid conditions in PM false alarms, t<1.

Consistent with previous research (e.g., Loft & Remington, 2010; Loft, Smith et al., 2011), the analysis of PM error reported above included all target events, even if the content of PM instructions was not recalled post-test. Post-test recall of targets can be influenced by whether the target received a PM response earlier and thus had an additional rehearsal, and for this reason post-test recall is not a pure measure of retrospective memory for PM instructions (Smith, Bayen, & Martin, 2010). Nonetheless, because the recall of the content of PM instructions was lower than in Experiment 1 and previous studies, we reanalysed PM errors after excluding individuals who did not correctly recall the PM instruction post-test. There were fewer PM errors made in the external aid conditions compared to the no aid condition, χ² (1) = 27.84, p < .001, ϕ² = .46. When including only participants who correctly recalled the PM task requirements, 15 out of 43 participants (35%) in the no aid condition failed to make the PM response, compared to two out of 90 participants in the external aid conditions (2%). One participant out of 44 in the non-adjacent aid condition failed to make the PM response, and one participant out of 46 in the adjacent aid condition failed to make the PM response.

To summarize the PM error results, under conditions were we presented a single PM target a significant time (39–42 min) after the presentation of a single PM instruction, we continued to find that external aids reduced PM error and false alarms compared to when no aids were provided, irrespective of whether the PM instruction was correctly recalled post-test. Whether the external aid was presented adjacent or non-adjacent to target aircraft had no effect on PM error or false alarm rate.

PM target acceptance time

Only participants who made the PM response to the target aircraft were included. In contrast to Experiment 1, there was a non-significant trend for participants provided external aids to be slower to accept target aircraft than participants in the no aid condition, t(119) = 2.13, p = .07, Mdiff = 0.91sec. Replicating Experiment 1, participants in the adjacent aid condition were faster to accept target aircraft than participants in the non-adjacent aid condition, t(89) = 2.34, p = .04, d = .49, Mdiff = 0.95sec.

Non-PM air traffic management

The non-PM air traffic management data are presented in Table 1³.

Conflict detection

The number of conflicts missed by participants in the no aid condition did not differ from the number of conflicts missed by participants in the control condition, t(93) = 1.87, p = .19. There was no difference in the number of conflicts missed when comparing the non-adjacent aid condition to the no aid condition, t < 1. The adjacent aid and non-adjacent aid conditions also did not differ in conflict detection misses, t < 1. Participants made an average of 0.63 conflict detection false alarms over all the test trials, with no effects approaching significance (all ps > .32). The same set of planned comparisons as Experiment 1 conducted for the conflict detection response time data revealed no significant differences between conditions (all ts < 1). Thus, there were no costs to conflict detection accuracy or response time.

Non-target acceptance time

Non-target acceptance times greater than 3 SDs from a participant’s grand mean were excluded (2.2% of acceptance times). Participants in the no aid condition were slower to accept non-target aircraft than participants in the control condition, t(93) = 3.38, p = .004, d = .70, Mdiff = 0.51sec. Participants in the non-adjacent aid condition were faster to accept non-target aircraft compared to participants in the no aid condition, t(93) = 2.64, p = .03, d = .54, Mdiff = 0.40sec, and compared to participants in the adjacent aid condition, t(93) = 2.66, p = .03, d = .55, Mdiff = 0.35sec.

Non-target hand-off times

Non-target hand-off times greater than 3 SDs from a participant’s grand mean were excluded (1.95% of hand-off times). Participants in the no aid condition were slower to hand-off aircraft than participants in the control condition, t(93) = 2.75, p = .02, d = .55, Mdiff = 0.40secs. No other differences between conditions reached significance, all ts<1.

Summary of Experiment 2

Experiment 2 makes several significant contributions. First, despite presenting only one target aircraft a significant time after the PM instruction, costs to non-PM air traffic management, in terms of slowed aircraft acceptance and hand-off response times, were demonstrated in Experiment 2 with effect sizes similar to those seen in Experiment 1 and in other previous studies that presented multiple PM instructions and multiple target aircraft (Loft, Finnerty et al., 2011; Loft, Pearcey et al., 2011; Loft & Remington, 2010; Loft, Smith et al., 2011). Second, we replicated the finding in Experiment 1 that the placement of external aids had a differential impact on PM air traffic management versus non-PM air traffic management; participants in the non-adjacent condition were faster to accept non-target aircraft compared to participants in the adjacent aid condition, whereas the use of adjacent aids resulted in faster PM target responses compared to the use of non-adjacent aids. Third, while many participants (38%) in the no aid condition failed to make the correct PM response, both adjacent and non-adjacent external aids greatly minimized this PM error.

General Discussion

External aids that support PM performance while also improving air traffic management have important potential for increased safety and decreased workload. We extended previous research in several ways. We demonstrated that the location in which the external aids appeared on the display had a differential impact on PM air traffic management versus non-PM air traffic management. As predicted, adjacent aids facilitated PM air traffic management compared to non-adjacent aids. However, in contrast to our predictions, non-adjacent aids reduced costs to non-PM air traffic management compared to adjacent aids. We also extended previous research by showing that external aids continued to be effective for reducing PM error when only a single PM target aircraft was presented, and that costs to non-PM air traffic management could continue to be observed under these task conditions. Finally, in addition to providing a full replication of Loft, Smith et al.’s (2011) study of external aids, we introduced new measures of non-PM air traffic management that take into account visual scanning and anticipation strategies.

Theoretical Implications

As just noted, the use of adjacent aids decreased the time taken to accept target aircraft compared to the use of non-adjacent aids. In addition to providing temporal context for the PM task, adjacent aids provided diagnostic information regarding the location of target aircraft, drawing attention to the PM target aircraft (Loft, Finnerty et al., 2011), increasing the likelihood that the external aid and target aircraft were processed as a single object (Awh et al., 2001). In contrast, when the non-adjacent aid was detected the focus of processing was shifted to a different part of the visual field and the participant needed to engage in visual reorienting activities in order to detect the target aircraft flashing at the sector boundary. However, while the adjacent aid conditions had faster PM responses relative to the non-adjacent aid condition, the overall effect of external aids on PM target response times varied across the two experiments. In Experiment 1, participants using external aids were faster to accept target aircraft than were participants in the no aid condition, but this difference trended strongly in the opposite direction in Experiment 2. Table 1 indicates that this change in the pattern across experiments is due to an increase in target acceptance times for the external aid conditions from Experiment 1 to Experiment 2, while the target acceptance times for the no aid condition remained fairly stable. In Experiment 2, when the external aids were only presented once and after significant delay, the decision processes required to first realize the significance of the external aid (see Brenesier & McDaniel, 2006; Smith, 2008) may have been more effortful, leading to longer target aircraft acceptance times in the external aid condition relative to Experiment 1.

Surprisingly, in both experiments the costs to non-PM air traffic management was smaller for participants that received the non-adjacent aids compared to participants that received the adjacent aids. Based on research into attentional capture, we predicted that adjacent aids would in fact reduce costs to non-PM air traffic management compared to non-adjacent aids because they were presented in a relevant region of the display and thus conformed to the attentional set (cf. Folk et al., 1992, 1994) that participant’s likely use for the routine tasks of aircraft acceptance, aircraft hand-off, and conflict detection. What does the unexpected pattern of non-PM air traffic management data say about the role of relevance and the placement of information in continuous, multitasking environments, such as air traffic control? One possibility is that more cognitive resources are required in preparation for the detection of adjacent aids because of the very fact that they are presented in closer proximity to the onset of other ATC events than non-adjacent aids are. That is, the presence of adjacent aids in the relevant display area must be distinguished from other air traffic events, such as aircraft appearing at sector boundaries, aircraft flashing for acceptance, and aircraft changing color when they violate minimum separation. This may partly be overcome in future research by presenting the adjacent external aid in a different color than the color that aircraft flash for acceptance. Another possibility is that the presentation of adjacent aids strongly associates them with the routine task of processing aircraft approaching or flashing at the sector boundary. In this way, adjacent aids may cause the PM task goal to be treated as a more primary task goal.

A third possibility is that in the adjacent aid condition, the nature of the adjacent aid served to redefine the characteristics of the PM target. Because the adjacent aids appeared with the target aircraft, a target could now be defined in terms of the presence of the aid, rather than by reference to a particular characteristic of the aircraft. In contrast, the non-adjacent aid clearly serves as a separate memory aid rather than a redefinition of what constitutes a target aircraft. Thus, participants in the non-adjacent aid condition may have been more likely to view the non-adjacent aid as an effective cue for signalling the presence of a target aircraft, which in turn would make participants less likely to allocate resources to the PM task prior to the external aid onset, reducing cost to non-PM air traffic management in the non-adjacent aid condition relative to the adjacent aid condition. Regardless of the ultimate underlying cognitive mechanism, the data indicate that the placement of external aids is reliably associated with differential benefits to PM related and non-PM related air traffic management tasks.

We found no evidence of costs to conflict detection for PM conditions in Experiment 2 where only a single target aircraft was presented. These null effects stand in contrast to the costs to conflict detection response time demonstrated in Experiment 1 and in previous studies (e.g., Loft, Finnerty et al., 2011; Loft & Remington, 2010), but should be interpreted with caution given the one-second timing resolution. In addition, participants in the PM conditions of Experiment 2 demonstrated costs to non-target aircraft acceptance and hand-off. Thus, participants were dedicating cognitive resources to the PM task (Savine et al., 2012; Smith, 2008) under conditions where a single target event was presented a significant time after the presentation of a PM instruction, a finding that has implications for the basic PM literature. Hicks, Marsh and Cook (2005) suggest that participants will make decisions about the extent to which resources need to be devoted to the PM task and will then establish an attentional allocation policy at the outset of the ongoing task. In contrast, Loft et al. (2008) have shown that when no targets appear during the ongoing task costs decrease, suggesting that attentional allocation policies can be adjusted during the ongoing task and that costs are maintained in part by the appearance of targets. In contrast to the Loft et al. findings, the costs to two of the three non-PM air traffic management measures were similar across Experiments 1, with multiple targets throughout the ATC task, and Experiment 2, in which a single target appeared only once towards the end of the ATC task. The different patterns in the current study and the Loft et al. study may be due to the fact that the current ongoing ATC task occurred in four blocks of two 5 minute trials, while the Loft et al. study used a continuous block of lexical decision trials. The PAM theory (Smith, 2003, 2008) proposes that at points of transition (e.g., when completing one task interval and starting a new task interval) we make decisions on whether to devote resources to the PM task during the upcoming task interval. The current study, with its multiple blocks and segmented trials, may have encouraged participants to make these attention allocation decisions at the start of each trial. Given that participants did not know when the target would appear, the PM task remained relevant and they therefore would be more likely to reallocate attention to the PM task throughout the ATC task. In contrast, while participants may have allocated resources to the PM task at the outset of the Loft et al. ongoing task, the continuous ongoing task in Loft et al. did not provide any points in which participants would be likely to reassess the relevance of the PM task and therefore did not provide opportunities to reallocate resources to the PM task.

Despite these observed costs to non-PM air traffic management tasks in Experiment 2, the fact that 38% of participants in the no aid condition still failed to deviate from the routine acceptance response indicates that the extent to which participants in the no aid condition were devoting resources to the PM task was insufficient to perform the PM task. This also indicates that in the absence of an external aid, attempts to further reduce PM errors would require substantial allocation of resources to the PM task, which in turn would result in an unacceptable additional increase in cost to non-PM air traffic management. In contrast, when external aids were provided in Experiment 2 the PM error rate remained near floor (4%), and costs to non-PM air traffic management were either reduced (non-adjacent aids) or at least remained equal (adjacent aids) compared to when no aids were provided.

Practical Implications, Limitations, and Conclusions

Previous work by Loft, Smith et al. (2011) indicated that external aids were only effective in reducing PM error if designed to command attention at the time that deviation from routine was required. This cast doubt over using static aids such as notepads and “post-it” notes in operational settings. Our interviewed controllers agreed with this conclusion, but also argued that external aids would be more effective if presented adjacent to target aircraft. The current data provide mixed support for these expert controllers’ intuition. While the efficiency of PM air traffic management was improved (i.e., faster PM target acceptance times) with the use of adjacent aids compared to non-adjacent aids, the costs to non-PM air traffic management were greater with the use of adjacent aids compared to non-adjacent aids. In operational settings, the choice of whether to present external aids adjacent or non-adjacent to target aircraft may depend on the relative importance of the efficiency of PM air traffic management versus non-PM air traffic management. Overall, these data highlight the fact that intuitions about our own cognitive processing in the absence of direct experience are fallible, and what seem to be good ideas, even to experts, may not prove so under the close scrutiny of laboratory testing. Since the expert controllers interviewed were not experts using the external aids being tested, they had no experience to guide their intuitions. Thus, while subject matter expert opinions on new tools can be valuable, final vetting of such tools must be based on evidence from well-controlled and ecologically valid empirical paradigms and follow up field testing.

Controllers’ intuitions of greater difficulty in remembering to perform deferred actions when there is a significant time period between PM encoding and target presentation, a condition they do know from their operational experience, proved to be correct. The high (38%) PM error rate when no aids were provided suggests that it is crucial to provide external aids under operational conditions where targets are only present once or twice over several hours, and indicates that controllers not provided external aids may potentially have to devote substantial resources to the PM task at the expense of other ATC tasks in order to remember to respond correctly to infrequent target events. In addition, the delayed speed of target acceptance in Experiment 2 compared to Experiment 1 with the use of external aids suggests controllers may take considerable time to recognize the relevance of an external aid onset if these external aids are presented only once or twice over several hours.

However, as noted elsewhere (e.g., Loft Smith et al., 2011), we cannot conclude that our findings with relatively inexperienced participants will necessarily generalize to field operations involving experienced controllers. For example, experienced controllers with well-defined performance standards may be less likely to forget intentions as often as that observed in Experiment 2. Nonetheless, given the number of aircraft and concomitant controller actions required each day, even small changes in error probabilities with an increased PM retention interval may translate into large differences in incidents (Dismukes, 2012; Shorrock, 2005). Furthermore, every such incident has the potential to translate into fatalities. On 3 September 2009 a Boeing 737 and 777 with combined total of 443 passengers nearly collided because a controller failed to remember to re-evaluate the aircraft pair (ATSB, 2010).

There also undoubtedly exist many practical constraints to consider before implementing external aids in ATC or other work settings, such as the fact that external aids in operational settings are unlikely to be 100% reliable. There exists a potential for external aids to be distracting. For example, the non-adjacent alert is like a heads-down display in that when it captures attention, the focus of processing is shifted to a completely different part of the visual field. In this case, the operator must engage in reorienting activities, perhaps even including eye and head movements, and this may be too costly to ATC operations. Further studies involving the analysis of the demands of operational contexts and new technologies (e.g., Next Gen) are required to fully understand the impact of PM alerts similar to ours.

In summary, operators in work contexts where goal-directed processing must play itself out in the presence of demanding perceptual display (e.g., ATC, submarine track management, seaboard navigation, pilot cockpit monitoring) are often required to remember to substitute atypical intended actions for routine actions. Given the potentially fatal consequences of failing to complete tasks in these work contexts the design of external aids to facilitate performance is crucial. We have continued to successfully apply basic psychological theory to understand the process underlying PM in simulated ATC and to examine how display tools may be designed to prevent error while simultaneously reducing costs to air traffic management.