Short-term memory stores organized by information domain

Abstract

Vision and audition have complementary affinities, with vision excelling in spatial resolution and audition excelling in temporal resolution. Here, we investigate the relationships among visual and auditory modalities and spatial and temporal short-term memory (STM) using change detection tasks. We created short sequences of visual or auditory items, such that each item within a sequence arose at a unique spatial location at a unique time. On each trial, two successive sequences were presented; subjects attended to either space (the sequence of locations), or time (the sequence of inter-item intervals), and reported whether the patterns of locations or intervals were identical. Each subject completed blocks of unimodal trials (both sequences presented in the same modality) and crossmodal trials (sequence 1 visual and sequence 2 auditory, or vice versa) for both spatial and temporal tasks. We found a strong interaction between modality and task: spatial performance was best on unimodal visual trials, while temporal performance was best on unimodal auditory trials. The order of modalities on crossmodal trials also mattered, suggesting that perceptual fidelity at encoding is critical to STM. Critically, there was no cost attributable to crossmodal comparison: in both tasks, performance on crossmodal trials was as good or better than on the weaker unimodal trials. STM representations of space and time can guide change detection in either the visual or the auditory modality, suggesting that temporal or spatial organization of STM may supersede sensory-specific organization.

Short-term (or working) memory is the remarkable ability to observe and encode information from the surrounding environment, maintain an internal representation of that information, and then draw on that memory representation to make judgments about future events (Voytek & Knight, 2010). An extensive previous literature has described the capacity and precision of visual short-term memory (STM; for reviews see Brady, Konkle, & Alvarez, 2011; Luck & Vogel, 2013; Ma, Husain, & Bays, 2014), verbal span (reviewed in Hurlstone, Hitch, & Baddeley, 2014), and non-verbal auditory memory (e.g., Parmentier & Jones, 2000; Visscher, Kaplan, Kahana, & Sekuler, 2007). Several groups have compared perceptual and memory effects in the visual and auditory modalities (e.g., Balch & Muscatelli, 1986; Guttman, Gilroy, & Blake, 2005; Tremblay, Parmentier, Guérard, Nicholls, & Jones, 2006; Shih, Kuo, Yeh, Tzeng, & Hsieh, 2009; McAuley & Henry, 2010), but only a few researchers have investigated how easy it is to use information encoded in one sensory modality to make judgments about information presented in another modality (Collier & Logan, 2000; Harrington, Castillo, Fong, & Reed, 2011). Descriptions of crossmodal judgment are valuable because they shed light on processes by which sensory-specific episodic information is transformed into more abstract representations, accessible to multiple perceptual streams. Here, we investigate short-term memory performance on spatial and temporal tasks for information presented in visual and auditory sensory modalities, and explore whether there are costs to STM performance when comparisons are across modality.

Specialization of sensory modalities

Sensory modalities are specialized for different information domains, as well as for different forms of environmental energy. Visual input is intrinsically spatial, comprising a two-dimensional array of luminances across the retina, and vision has a corresponding affinity for spatial information. Spatial characteristics define the features to which the visual system is most sensitive, such as edges and texture (Hubel & Wiesel, 1959). Spatiotopic cortical maps are found throughout the visual processing pathway (Swisher, Halko, Merabet, McMains, & Somers, 2007; Silver & Kastner, 2009). Auditory perception of space, on the other hand, is relatively poor (Jackson 1953), and spatiotopic maps have not been identified at any stage of the mammalian cortical auditory pathway (Barrett & Hall, 2006).

Auditory sensory inputs comprise a series of fluctuating pressure oscillations; it is the frequency of these oscillations and how the energy at each frequency changes through time that convey information to the auditory sense. Audition has an affinity for temporal information, with exquisite sensitivity to temporal features such as onset and frequency, and primary auditory cortex displays a tonotopic organization (Merzenich & Brugge, 1973; Wessinger, Buonocore, Kussmaul, & Mangun, 1997; Da Costa et al. (2011). The auditory system is able to make very fine discriminations in the judgment of time intervals (Creelman, 1962; Lhamon & Goldstone, 1974; Ivry & Hazeltine, 1995), with precision at least an order of magnitude better than in the visual modality (Lhamon & Goldstone, 1974).

These complementary sensory affinities have perceptual consequences, often described as modality appropriateness (Welch & Warren, 1980). Vision tends to dominate when the cognitive task at hand is spatial. For instance, when reporting the location of an auditory stimulus source, observers are biased towards the spatial location of a visual stimulus presented at a similar time, a phenomenon known as the ventriloquism effect (Howard & Templeton, 1966; Bertelson & Aschersleben, 1998). Conversely, audition tends to dominate when the cognitive task is temporal, resulting in a temporal ventriloquism effect. When reporting the time at which a visual event occurred, participants are biased towards the time (temporal location) of a nearby auditory stimulus (Shams, Kamitani, & Shimojo, et al., 2000; Fendrich & Corballis, 2001; Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Burr, Banks, & Morrone, 2009; Bizley, Shinn-Cunningham, & Lee, 2012). Welch and Warren (1980) hypothesized that these effects arose from the innate correspondence between sensory modalities and information domains. More recently, these effects have been attributed to each modality’s reliability in the information domain at hand. If the reliability of input in one sensory modality is reduced, that modality’s contribution to the final percept is also reduced; analyzed based on the reliability of information, the brain comes remarkably close to optimally combining information across sensory modalities (Alais & Burr, 2004).

Similar effects of modality occur in STM. Memory for spatial locations is superior when those locations are presented visually rather than through sound (Lehnert & Zimmer, 2008), and abstraction of the up/down contour of a sequence of sensory events is more efficient from visuospatial stimuli than from melodic stimuli (Balch & Muscatelli, 1986). On the other hand, memory for rhythms and time intervals is superior when stimuli are presented to the ears rather than the eyes (Collier & Logan, 2000). Indeed, auditory distractors interfere with memory for visual rhythms, hinting that both may be held in a common store (Guttman et al., 2005). The perceptual reliability of visual and auditory inputs may shape short-term memory for spatial and temporal information.

Nature of short-term memory stores

The modal model of STM is Baddeley and Hitch (1974)’s working memory description, in which discrete visuospatial and verbal stores are both linked to a central executive component. Early evidence for this separation of two stores came from studies that crossed visuospatial or verbal STM tasks with concurrent visuospatial or verbal secondary tasks. These studies consistently identified modality-specific interference effects. For example, STM for visual imagery, but not for verbal lists, is disrupted by a simultaneous visual pursuit task (Baddeley, Grant, Wight, & Thompson, 1975), or a spatial manipulation task (Logie, Zucco, & Baddeley, 1990; Guérard & Tremblay, 2008; Alloway, Kerr, & Langheinrich, 2010). Conversely, maintenance of auditory STM is impaired by simultaneous verbal articulation, while visual STM is not (Kroll, Parks, Parkinson, Bieber, & Johnson, 1970; Guérard & Tremblay, 2008; Alloway et al., 2010), and verbal STM, but not visuospatial STM, is disrupted by a simultaneous arithmetic task (Logie, Zucco, & Baddeley, 1990). Further evidence for two discrete stores comes from distraction effects. Unexpected, task-irrelevant events in the spatial or verbal information domain impair spatial or verbal memory performance and reasoning, respectively (Farmer, Berman, & Fletcher, 1986; Lange, 2005). Syncopated tapping also disrupts verbal STM, suggesting that rhythm draws on the verbal STM store (Larsen & Baddeley, 2003). Additional support for the separate-stores account comes from lesion patients with impaired digit span but normal spatial span, and other patients with the converse pattern of impairment (Vallar & Baddeley, 1984; Hanley, Young, & Pearson, 1991).

Previous work on non-verbal auditory information and STM is sparser. Verbal STM is disrupted by vocal and non-vocal music, but not by auditory noise, implying that some non-verbal auditory information shares a STM store with verbal information (Salamé & Baddeley, 1989). Memory for pitch is disrupted by intervening tones or words of similar pitch (Deutsch, 1972; Semal, Demany, Ueda, & Hallé, 1996), but not by visual patterns (Pechmann & Mohr, 1992). Memory for timbre is also disrupted by similar intervening items (Starr & Pitt, 1997). Non-verbal auditory information is thus believed to share a STM store with verbal information, separate from visuospatial information (Baddeley, 2012). Intriguingly, visual and auditory rhythms seem to share a STM store, although performance when judging auditory rhythms is superior (Collier & Logan, 2000).

A few studies have pointed to a shared memory process that is recruited by both visual and auditory or verbal STM. Verbal STM is impaired for words that are visually and phonologically similar compared to words that are phonologically similar but visually different (Logie, Della Sala, Wynn, & Baddeley, 2000). Visual and auditory STM are influenced by many of the same factors, such as serial position, retention interval, and load (Visscher et al., 2007). STM capacity for mixed auditory and visual arrays is larger than capacity for either visual or auditory, but smaller than their sum, suggesting that some portion of STM capacity is shared between modalities (Saults & Cowan, 2007; Fougnie, Zughni, Godwin, & Marois, 2014).

Crossmodal memory-guided cognition

In daily life, humans frequently work with information in multiple sensory modalities. We use visual information to predict forthcoming auditory input, and vice versa. For example, a person standing at the kitchen sink might hear a noise behind them, and, on turning around, expect to see a family member at the kitchen door. A different person or a different location would be quite unexpected. Only a few studies have investigated memory performance when using visual stimuli to probe auditory encoding, or vice versa (crossmodal memory). Both examples that we know of are investigations in the temporal domain, one into representation of single intervals (Harrington et al., 2011), the other into representation of rhythms (Collier & Logan, 2000). Both studies found that subjects’ performance on unimodal auditory conditions was superior to performance on unimodal visual conditions. Crossmodal comparison of single intervals is better than unimodal comparison when the auditory interval is presented first, but worst than unimodal comparison when the visual interval is presented first (Harrington et al., 2011), while crossmodal comparison of rhythms is neither consistently better nor consistently worse than unimodal comparison, but somewhere in between (Collier & Logan, 2000).

The present work

Here, we measure short-term memory for visual or auditory stimuli using a sequence change detection task that requires storage of either spatial locations or temporal intervals. Subjects performed both unimodal change detection (comparing either two visual sequences or two auditory sequences) and crossmodal change detection (comparing a visual to an auditory sequence, or vice versa). We tested four hypotheses:

• H1: STM modality appropriateness. The visual and auditory modalities differ in their ability to extract and represent spatial and temporal information. Unimodal change detection performance for spatial tasks will be superior for visual trials compared to auditory trials, while performance on temporal tasks will show the opposite effect.

• H2: Shared domain-specific memory stores. If STM representations are stored according to task demands (i.e., information domain, rather than input modality) and these task-based STM stores can be accessed by both modalities, then performance on crossmodal change detection will be as good or better than performance in the weaker unimodal case. That is, costs of crossmodal STM come only from modality-specific limitations, rather than from any additional cost of translating between modalities.

• H3: Dependence on STM encoding. Observers likely minimize STM load by encoding only the first sequence into STM; they then perform an online comparison between the memory representation and the incoming second sequence to determine whether each sequence element matches the memory representation. If so, the fidelity of the STM representation of the first sequence is crucial, dominating performance. In crossmodal tasks, performance on spatial tasks will be better on visual-first trials than on auditory-first trials; performance on temporal tasks will be better on auditory-first than visual-first trials.

• H4: Interference from task-irrelevant changes. If spatial and temporal information domains interact, it may be easy to detect that a change occurred, but difficult to discriminate changes in timing from changes in spatial location. Some authors have hypothesized that either location (e.g. Treisman & Gelade, 1980; Johnston & Pashler, 1990) or time of occurrence (e.g. Dixon & Spitz, 1980; Howard & Kahana, 2002) is an essential cue for binding features in different modalities or different information domains into a coherent percept and a shared memory representation. Therefore, if there are task-irrelevant changes between the first and second sequence (i.e., while subjects are performing the spatial task, sequence timing changes), it will disrupt the ability to detect change in the task-relevant information domain.

Our results supported H1, H2, and H3, but not H4.

Methods

All protocols were approved by the Institutional Review Board of Boston University.

Subjects

Twenty-five young adults enrolled in Introduction to Psychology at Boston University participated in this study for partial course credit (ages 18–22, 16 women). We computed mean change detection sensitivity, which we used to screen out participants with very low performance. We a priori selected an arbitrary threshold of d′ = 0.5 and five subjects whose performance was below this threshold were excluded. All results presented here are from the remaining 20 subjects. Our sample size (20 subjects passing a minimum-performance criterion) was selected a priori using effect size estimates from pilot data.

Experimental task

We adapted the change detection paradigm (Phillips 1974; Vogel, Woodman, & Luck, 2001) to require subjects to compare two short sequences. Subjects first observed a sequence in one modality (visual or auditory), followed by a brief delay (0.5 s). After the delay, subjects observed a second sequence and were asked to report whether it was identical to or different from the first in either its sequence of spatial locations (Space task) or its sequence of temporal intervals (Time task). Note that the properties of the stimuli were identical in both tasks; only the instructions given to the subjects differed.

Each subject performed both the Space and the Time tasks in both unimodal and crossmodal forms. On each trial of unimodal change detection, subjects encountered either two visual sequences or two auditory sequences, and decided whether the sequences were identical in the task-relevant information domain (either Space or Time). We will denote these unimodal trials as VV and AA trials, respectively. On each trial of crossmodal change detection, subjects encountered two sequences in different modalities, either visual first and auditory second or vice versa, and compared across modalities to determine whether the two sequences matched in the task-relevant information domain. We will denote these types of trials as VA and AV trials, respectively.

A third factor was the presence or absence of changes in the task-irrelevant information domain, allowing us to test whether such changes affected performance. That is, while performing the Space task, were subjects impaired by irrelevant changes in the Time domain? To investigate this, we constructed blocks of trials in which the task-irrelevant information domain was always fixed between the first and second sequence of each trial, and blocks of trials in which the task-irrelevant information domain had a 50% probability of containing a change. The order of blocks within the Space and Time tasks was randomized for each subject.

Stimuli

Visual and auditory stimuli were constructed to contain both spatial and temporal information. Each item in a sequence had a unique spatial location; each pair of items was separated by a unique temporal interval. Figure 1 shows schematics of the visual and auditory stimuli used in this experiment.

Figure 1.

Visual and auditory stimuli for the sequence change detection task. (A) Visual stimuli comprised an array of mostly static images. Across the ensuing sequence, four of these images were replaced by their mirror-flipped counterparts (one at a time). When performing the Space task, subjects were to remember the sequence of locations of these four replacements; when performing the Time task, they were to remember the sequence of intervals between successive replacements. (B) Auditory stimuli comprised a series of 50 ms complex tones, lateralized by interaural time difference. When performing the Space task, subjects were to remember the sequence of lateral locations; when performing the Time task, they were to remember the sequence of intervals between successive tones.

For visual sequences (Figure 1A), subjects observed a semicircular array of five images presented on a CRT monitor. Each sequence item consisted of the instantaneous left-right mirror flip of one image. An item thus had a spatial location within the array, and a temporal interval separating it from the previous flip. The images were drawn from a collection of animal images (e.g., drawings, photographs) against white backgrounds, retrieved from a web search, and scaled so that the images were clearly visible and approximately the same size. On each trial, five copies of a single image were used. Whether each copy was in the original orientation or mirror-flipped at initial onset was randomly assigned, so that the instantaneous orientations of the images provided no information about whether or not that item had already flipped. This stimulus paradigm was originally designed for a long-term memory experiment and was adapted for the present short-term memory purposes; here, the identity of the animals is irrelevant, but the animal images may have increased subject interest and vigilance in the task.

These stimuli are not perfectly parallel to their auditory equivalents (described below). All five images were visible on the screen throughout the entire sequence, while auditory items were presented one at a time. We had three reasons for this decision. First, sudden, discrete onsets and offsets of visual stimuli are unusual among natural visual inputs, whereas changes in state of an already-present item are more common. Second, in piloting, we found that onsetting and offsetting visual stimuli drove a strong subjective experience of an internal auditory representation, regardless of the task. Finally, the persistent images acted as visual masks, making the visual tasks more attentionally demanding. For auditory sequences (Figure 1B), subjects listened to a stream of complex tones. Tones were presented via stereo headphones at a subject-adjusted, comfortable, clearly audible listening level (between 65 and 85 dB SPL). A sequence item consisted of a single tone, lateralized by interaural time difference. An item thus had a spatial location in azimuth, and a temporal interval separating it from the previous tone. We generated complex tones by first selecting one of ten base frequencies (logarithmically spaced between 200 Hz and 400 Hz). Two other frequencies were then selected at 1.33 and 1.59 times the base frequency. Tones included these three fundamental frequencies and their first three harmonics at equal intensity, and were windowed with a 16 ms cosine-squared ramp at onset and offset. This process generated a set of tones with similar timbre and varying pitch. All tones within a trial were composed of the same frequencies. Each tone had an interaural time difference drawn from the set of −1000 μs, −350 μs, 0 μs, 350 μs, and 1000 μs (selected randomly as noted below), producing tones whose lateral angle corresponded roughly to the five spatial locations in the visual display.

Stimulus sequences in both modalities comprised four sequence items with three inter-item intervals. At the start of each trial, a sequence of locations was drawn without replacement from the set of far-left, left, center, right, and far-right locations. Similarly, a sequence of inter-item intervals was drawn from the set of 250.0 ms, 404.5 ms, 654.5 ms, and 1059.0 ms intervals. This order of locations and intervals specified that trial’s first sequence.

This family of intervals was selected to comply with three constraints. We selected intervals that were not integer multiples of each other, to reduce rhythmicity of the stimuli. Then, we selected intervals that were below approximately 1 s in duration, as longer interval timing disrupts performance (Collier & Logan, 2000). Finally, pilot testing showed that intervals shorter than about 250 ms are nearly impossible to track in the visual modality.

During blocks of trials in which the task-irrelevant information domain was fixed, the second sequence was either identical to the first or, with 50% probability, contained a change in the task-relevant information domain, with two locations or two intervals swapping serial position (with the stipulation that the first item’s location never changed, in an attempt to equate the memory load between the Space and Time tasks). During blocks of trials which included task-irrelevant changes, such changes occurred with 50% probability and were independent of the presence or absence of a change in the task-relevant information domain. The second sequence could thus contain changes in locations, intervals, both, or neither, each with 25% probability.

Design and procedures

As described above, this study had a 2 (task: Space, Time) x 4 (modality permutation: VV, VA, AV, AA) x 2 (task-irrelevant changes: yes, no) within-subjects design. Pilot data suggested that intermixed conditions dramatically impaired subjects’ ability to discriminate changed from unchanged trials, possibly due to the demands of shifting cognitive set. We therefore used a blocked design in our experiment. Each subject completed one block (48 trials) of each condition. Response identities and reaction times were recorded; because subjects were not prompted to answer until after the second sequence was complete, the reaction times are minimally informative and only the response identities (i.e. “same” or “different”) are analyzed here.

After giving informed consent and completing a basic demographics questionnaire, subjects received instruction on either the Space task or the Time task. They then completed all eight blocks of trials in that task. After a brief break, subjects received instruction on the remaining task, and again completed eight blocks of trials in that task. The order of tasks was counterbalanced across subjects; the order of conditions within each task was randomized for each subject.

Performance was operationalized as change detection sensitivity, measured by d′. We chose to measure sensitivity rather than accuracy because it is not affected by subjects’ criteria for reporting “different,” and we suspected that those criteria might differ between conditions.

Results

Subjects were able to discriminate changed from unchanged sequences (overall mean d′ = 1.02, SD = 0.26). A two-way repeated-measures ANOVA with factors modality permutation (VV, VA, AV, and AA) and task (Space and Time) identified a significant main effect of modality (F(3,57) = 13.47, p < .0001), a significant main effect of task (F (1, 19) = 7.76, p = .012), and a significant interaction between them (F (3, 217) = 69.34, p < .0001).

Figure 2 shows performance on the Space and Time tasks for each modality permutation, collapsed across blocks where the task-irrelevant information domain was fixed and blocks where it could change. As we predicted, there is a substantial STM modality appropriateness effect, with performance on VV higher than on AA for the Space task (t (19) = 12.30, p < .0001, Cohen’s d = 2.78), but lower for the Time task (t (19) = 7.64, p < .0001, Cohen’s d = 1.71).

Figure 2.

Change detection sensitivity on the Space task (left, in blue) and Time task (right, in orange) for all four modality permutations. Performance was best in the task-appropriate modality, and shows no cost of crossmodal change detection beyond that attributable to the perceptual weakness of the task-inappropriate modality.

*p < .05; **p < .01; ***p < .001.

The two crossmodal conditions are consistently as good or better than the weaker unimodal condition. In the Space task, performance on VA is better than on AA (t (19) = 3.32, p = .004, Cohen’s d = 0.76), and there is no significant difference between performance on AV and on AA (t(19) = −0.45, p = .66, Cohen’s d = 0.10). Similarly, in the Time task, performance on AV is better than on VV (t(19) = 2.90, p = .009, Cohen’s d = 0.66), and there is no significant difference between performance on VA and on VV (t (19) = 0.20, p = .84, Cohen’s d = 0.05). This is consistent with a STM store that can be freely accessed by both modalities, with no additional cost of switching between them.

Finally, in the crossmodal conditions, the order of modalities matters. In the Space task, there is a substantial advantage to encountering the visual sequence first (VA > AV, t(19) = 4.02, p = .0007, Cohen’s d = 0.93); in the Time task, there is a moderate advantage to encountering the auditory sequence first (AV > VA, t (19) = 2.41, p = .03, Cohen’s d = 0.54). This highlights the importance of STM encoding of the initial sequence in this paradigm.

Using multiple linear regression, we quantified the influence of task, modality, and task-appropriateness on change detection performance. To reduce effects of individual differences in memory capacity and motivation, we first standardized each subject’s d′ scores over the sixteen blocks of trials. Then we fit a linear model using five binary predictors: (β1 & β2) the task-appropriateness of the modalities in which the first and second sequences were presented; (β3) the task (Time > Space); (β4 & β5) the modalities of the first and second sequences (V > A). (Note that all predictors are binary, with mean 0.5.) The model also included a constant term.

Figure 3 shows the resulting beta weights. The single strongest predictor is whether the modality of the first sequence is appropriate to the task the subject is performing i.e. visual for the Space task and auditory for the Time task (β1 = 1.00, s.e. = 0.09). This factor alone accounted for 0.26 of the variance in standardized scores. Task-appropriateness of the second sequence’s modality was the next-strongest factor (β2 = 0.48, s.e. = 0.09), followed by task (a benefit for Time, β3 = 0.37, s.e. = 0.09). Effects of modality alone were modest. This analysis confirms and quantifies the striking degree to which task-appropriateness of the first sequence affects change-detection performance.

Figure 3.

Factor weights from a multiple linear regression comparing the effects of modality, task-appropriateness, and task. Alignment between task and the first sequence’s modality is by far the strongest predictor, accounting for a full standard deviation in performance.

We also assessed whether task-irrelevant changes between the encoding sequence and the probe sequence had any effect on behavior. Figure 4 shows performance on blocks of trials when the event values in the unattended information dimension were always the same in the two sequences (fixed within a block; top panel), as well as in the mixed blocks, broken down by whether the task-irrelevant information was fixed or changed (bottom panel). A three-way ANOVA with factors modality permutation, task, and task-irrelevant information (always fixed, fixed trials in mixed blocks, changed trials in mixed blocks) found no significant effect of task-irrelevant information (F(2, 38) = 0.96, p = .391), no interaction between task-irrelevant information and either modality permutation (F(2,38) = 0.55, p = .584) or task (F(6,114) = 0.97, p = .447), and no three-way interaction (F(6,114) = 1.29, p = .269). In short, we found no evidence for any effect of the stability or changeability of the task-irrelevant information domain. A multiple regression similar to the above that included whether each block’s task-irrelevant dimension was fixed or mixed resulted in very similar beta weights on all regressors, confirming this result.

Figure 4.

Change detection sensitivity on the Space task (left, in blue) and Time task (right, in orange) for all four modality permutations, broken down by the presence or absence of task-irrelevant changes. There was no effect of task-irrelevant changes.

Discussion

We developed a sequence change detection task that allowed us to probe unimodal and crossmodal short-term memory use of spatial and temporal information. In both spatial and temporal tasks, change detection substantially improved when stimulus sequences were presented in the modality that was most appropriate for the task demands: spatial task performance was best in the unimodal visual condition and temporal task performance was best in the unimodal auditory condition. We also showed that crossmodal change detection has no performance costs beyond those associated with the poorer perceptual representation in the weaker modality, suggesting that short-term memory stores for spatial and temporal information are accessible to both sensory modalities. Further, in crossmodal change detection, performance depended on the order of the modalities. Spatial change detection was better when the first sequence was visual and the second auditory than vice verse; temporal change detection showed the converse effect.

STM modality appropriateness

Demonstrating that audition and vision are uneven in their ability to represent spatial and temporal information is consistent with an extensive body of prior work, and confirms the validity of our task, providing confidence that we are probing spatial and temporal representations in STM. Temporal information is perceived more accurately and precisely through the auditory than the visual modality (Lhamon & Goldstone, 1974), and this advantage extends into short-term memory (Collier & Logan, 2000; Guttman et al, 2005). Conversely, spatial information is perceived more accurately and precisely through the visual than the auditory modality (Jackson 1953), and this advantage also holds in short-term memory (Lehnert & Zimmer, 2008). These results, along with this prior work, confirm that the modality appropriateness hypothesis (Welch & Warren, 1980) can be generalized to describe properties of STM.

Our results do not separate perceptual limitations in the weaker modality from STM limitations. We think it very likely that the impaired spatial change detection we observed in the unimodal auditory condition and the impaired temporal change detection we observed in the unimodal visual condition reflect, at least in part, those modalities’ reduced ability to detect and extract the relevant stimulus features. However, our further study of crossmodal change detection (e.g., AV and VA conditions) equalizes perceptual limitations and isolates STM processes, particularly retrieval and comparison.

Shared domain-specific memory stores

It was unclear how modalities and task demands would interact when the first and second sequences were presented to different modalities. If STM were stored strictly according to sensory modality (as in a naïve implementation of the Baddeley and Hitch, 1974 model), crossmodal change detection would require an extra step of translation from one modality-specific representation to another, which should result in additional costs to performance. If this were the case, crossmodal change detection would be even more difficult than change detection in the weaker unimodal condition. Conversely, if STM were stored according to the informational requirements of the task, although a task-appropriate modality may still have a perceptual advantage, there should be no costs of crossmodal comparison that are not attributable to limitations from the representation in the less accurate modality for that information domain. Our results support this latter interpretation, with both crossmodal conditions resulting in change detection that is as good or better than in the weaker unimodal condition (although still below the stronger unimodal condition). Our pattern of results matches that previously observed in STM for temporal sequences (Collier & Logan, 2000); here, we’ve shown that it holds for the spatial information domain as well.

One alternative interpretation of these data is that STM stores are strictly segregated according to sensory modality, but the brain can translate between sensory-specific representations with perfect fluency. Memory capacity measures in unimodal and crossmodal STM tasks may provide an avenue for distinguishing between these interpretations (Saults & Cowan, 2007; Fougnie & Marois, 2011). Nonetheless, we argue that memory stores that are fluently accessible to guide judgments in any modality are not meaningfully separated by modality in the classic sense. Information domain appears to supersede sensory modality as an organizational principle for STM stores.

We suspect that crossmodal change detection involves a process of encoding sensory input into a representation that is appropriate for the task demands. A number of studies have suggested that visual temporal processing relies on auditory mechanisms (e.g. Guttman et al., 2005; Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008; Keller & Sekuler, in press), although others dispute this (McAuley & Henry, 2010). Similarly, auditory spatial representations seem to rely on the visuospatial processing pathway (Martinkauppi, Rämä, Aronen, Korvenoja, & Carlson, 2000). A recent project investigating the brain networks engaged during unimodal change detection for spatial and temporal information identified visual-biased and auditory-biased areas in posterior lateral frontal cortex (Michalka, Kong, Rosen, Shinn-Cunningham, & Somers, 2015). Activity in the visual-biased areas was stronger in an auditory spatial task than in an auditory temporal task, suggesting that these areas may be recruited to support processing in an information domain that is better-suited to vision than audition. Auditory-biased areas, conversely, were more strongly activated in a visual temporal task than in a visual spatial task, again suggesting recruitment to support processing in the temporal information domain. Given this evidence for domain recruitment, it is likely that similar processes support crossmodal change detection.

Dependence on STM encoding

In crossmodal change detection, the modality of the first sequence had a strong impact. Subjects performed substantially better when the first sequence was presented in the task-appropriate modality than when the first sequence was task-inappropriate. In change detection tasks, subjects encode the first stimulus into STM, and then use the memory representation to drive a comparison process as they observe the second stimulus (Collier & Logan, 2000). In a sequential task such as ours, this comparison likely involves the sequential allocation of attention to particular locations at particular times (those locations and times that match the stored memory representation of the initial sequence). The stronger the memory representation (and thus expectations about forthcoming sequence items), the better the discrimination between same and different trials (Summerfield & Egner, 2009). If subjects were, instead, encoding the entirety of both sequences into memory and performing comparisons on the memory representations, we would not expect to see a large asymmetry between performance for visual-first and auditory-first crossmodal conditions.

There are several possible mechanisms that could give rise to the strong effect of task-modality appropriateness at the first stimulus sequence. One mechanism is the perceptual limitations of each sensory modality in its weaker information domain. That is, vision is relatively poor at extracting timing information from sensory input (Lhamon & Goldstone, 1974), and audition is relatively poor at extracting spatial information (Jackson 1953). The change-detection impairment that we observed when the first sequence was presented in the weaker modality may be attributable to confusions between locations (auditory Space task) or intervals (visual Time task). Another possible source is differences in modality-specific access to STM. It may be the case that, while both modalities can freely access STM stores, one has an advantage. This privileged access may result in stronger or more-precise memory representations when information is presented in the privileged modality (auditory for temporal information and visual for spatial information). Our experiment does not allow us to separate these two possible explanations; further work will be needed to test these effects.

A third possible explanation is that memory representations decay more quickly when the modality and the task are poorly aligned (Collier & Logan, 2000; Donkin et al., 2014). When the first sequence is presented in a task-inappropriate modality, its memory representation may have deteriorated (via either sudden death or gradual decay; Zhang & Luck, 2009) by the time that subjects are assessing whether the two sequences are the same. When the first sequence is presented in the task-appropriate modality, its memory representation may remain more intact, and more effectively comparable to the second sequence. We find this account less convincing than the perceptual limitations or privileged memory access explanations, as the timescales in our experiment are shorter than those generally identified to increase memory decay over time, but the study reported here is not sufficient to disprove this hypothesis. Further work measuring performance changes at longer or shorter timescales may be able to tease these two effects apart.

Interference from task-irrelevant changes

Contrary to our interference hypothesis, subjects were not noticeably affected by changes between sequence one and sequence two in the task-irrelevant information domain. That is, subjects were able to consistently separate temporal from spatial information in short visual and auditory sequences. Previous work has suggested that shared location may be an essential cue for binding multiple stimulus features into a single representation (e.g. Treisman & Gelade, 1980; Johnston & Pashler, 1990). Shared timing has similarly been proposed as a cue for binding, particularly across modalities (Dixon & Spitz, 1980) or as an organizing principle in memory (Howard & Kahana, 2002). Our results do not preclude location or timing playing an important role in perceptual and episodic binding; they merely highlight an ability to “unbind” and separate the domains. We do not have enough information to assess whether this separation occurs before memory encoding, so that only the relevant information domain is stored in memory, or whether it occurs at memory retrieval, with subjects selecting the relevant information from the memory representation in order to guide their comparisons. This appears to be a promising avenue for further investigation.

We should note that the classical tau effect (changes in the time intervals between successive stimuli bias the perceived spatial distance between them; Helson & King, 1931) and kappa effect (changes in the spatial distance between successive stimuli bias the perceived time intervals between them; Cohen, Hansel, & Sylvester, 1953) also predict interactions between task-relevant and task-irrelevant information domains. However, unlike those earlier studies, our task included a small number of discrete locations, which were explicitly identified to subjects, as well as a small number of discrete time intervals. This may have minimized bias in location or interval judgments.

General discussion

Our task required subjects to make at least one explicit translation between reference frames. The spatial locations in the visual task comprised five equally spaced locations around the perimeter of a semi-circle on the (vertical) screen in front of them; the spatial locations in the auditory task comprised five locations in the azimuth (horizontal) plane, with interaural differences selected to roughly match the angular separation of the visual locations (note that the mapping between interaural time differences and lateral angle is compressive, rather than linear, explaining our choice of binaural cues). No subjects reported confusion about this remapping; modern humans perform such tasks on a regular basis, such as mapping from a mouse or touchpad to a computer screen, and are quite fluent in them.

One key limitation of this study is that we only investigated a single family of intervals, both for the inter-item intervals within sequences and for the interstimulus interval between the two sequences of a given trial. Crossmodal memory likely requires some time to be recoded from a modality-specific representation to a modality-general one, or from a task-inappropriate to a task-appropriate representation (see Collier and Logan, 2000 for an extensive investigation of the effects of rate and timing on a crossmodal rhythm task). We suspect that if we reduced the time between paired sequences, or shortened the sequence intervals, performance on the task would be decreased, especially in visual timing tasks. Conversely, it is likely that if the intervals were lengthened to supra-second ranges, quite different results might occur. Collier and Logan (2000) found that at those longer intervals, rhythm change detection was generally impaired, and the advantage of auditory presentation was reduced.

To our knowledge, this is the first study simultaneously investigating effects of sensory modality and of task demands on change detection performance. Our results emphasize the importance of memory encoding, and the likelihood that sensory inputs are recoded into a task-specific rather than a modality-specific representation, as well as reaffirming the strong links between sensory modalities and specific information domains.