Auditory short-term memory in the primate auditory cortex

Abstract

Sounds are fleeting, and assembling the sequence of inputs at the ear into a coherent percept requires auditory memory across various time scales. Auditory short-term memory comprises at least two components: an active ‘working memory’ bolstered by rehearsal, and a sensory trace that may be passively retained. Working memory relies on representations recalled from long-term memory, and their rehearsal may require phonological mechanisms unique to humans. The sensory component, passive short-term memory (pSTM), is tractable to study in nonhuman primates, whose brain architecture and behavioral repertoire are comparable to our own. This review discusses recent advances in the behavioral and neurophysiological study of auditory memory with a focus on single-unit recordings from macaque monkeys performing delayed-match-to-sample (DMS) tasks. Monkeys appear to employ pSTM to solve these tasks, as evidenced by the impact of interfering stimuli on memory performance. In several regards, pSTM in monkeys resembles pitch memory in humans, and may engage similar neural mechanisms. Neural correlates of DMS performance have been observed throughout the auditory and prefrontal cortex, defining a network of areas supporting auditory STM with parallels to that supporting visual STM. These correlates include persistent neural firing, or a suppression of firing, during the delay period of the memory task, as well as suppression or (less commonly) enhancement of sensory responses when a sound is repeated as a ‘match’ stimulus. Auditory STM is supported by a distributed temporo-frontal network in which sensitivity to stimulus history is an intrinsic feature of auditory processing.

1. Introduction

Nonhuman primates navigate a rich auditory world in which sounds guide social interactions and signal environmental dangers. Their repertoire of acoustically driven behaviors suggests monkeys should perceive and remember sounds much as humans do. However, accumulating evidence suggests this is only half true: perceptually, hearing in macaques is comparable to our own (Jackson et al., 1999; Stebbins and Moody, 1994), but this does not seem to be true of auditory memory, a domain in which monkeys are slow to learn and their performance is consistently poor relative to that in the visual modality (Colombo and D'Amato, 1986; Colombo and Graziano, 1994; Cowey and Weiskrantz, 1976; Fritz et al., 2005; Kojima, 1985; Wright, 1999; Wright, 2007). Does this discrepancy represent a species difference between monkeys and humans, a modality difference between vision and audition, or some combination of both? How does the faculty of long-term memory (LTM) constrain possible models of short-term memory (STM), and their instantiation in the cortex? And in light of these discrepancies, is the nonhuman primate a valid model for the neural mechanisms of human audition?

This review summarizes recent work on auditory memory in the nonhuman primate and addresses outstanding questions about the application of these findings to human hearing. A key distinction will be the separation of auditory STM into two components: a sensory trace, which may be passively retained, and a phonological rehearsal mechanism, as would subserve active (or ‘working’) memory (Baddeley, 2003; Cowan, 1984; Cowan, 2008). These components may be mapped roughly to the ventral and dorsal streams (Hickok and Poeppel, 2007; Rauschecker and Scott, 2009) of auditory cortex, respectively. We will propose that nonhuman primates offer a tractable model for the perceptual and mnemonic processes of the ventral stream, including the trace that underlies passive STM (pSTM). By contrast, the contribution of the dorsal stream to auditory memory in humans has likely evolved in tandem with language, and may exist in nonhuman primates in only a rudimentary form.

2. Behavioral studies of auditory memory in monkeys

2.1 Do monkeys possess long-term auditory memory?

The nonhuman primate has been a productive model for the study of memory, establishing that long-term visual recognition relies upon cortical areas in the medial temporal lobe (MTL), particularly the rhinal cortex (Mishkin, 1982). This finding was soon extended to tactile memory (Murray and Mishkin, 1983), setting the expectation that the MTL may also support auditory LTM. To test whether selective ablation of the MTL disrupted auditory LTM, monkeys were trained to perform an auditory variant of the classic test for recognition memory, delayed-match-to-sample (DMS). Training monkeys to perform auditory memory tasks has been shown to be slow and arduous relative to training on visual tasks (Colombo and D'Amato, 1986; Colombo and Graziano, 1994; Cowey and Weiskrantz, 1976; Kojima, 1985; Wegener, 1964; Wright, 1999; Wright, 2007). This problem may have been exacerbated in earlier studies that used small stimulus sets, thereby creating confusion as to whether a sound was heard in the current trial or a previous one (Colombo et al., 1990; Colombo et al., 1996; the role of stimulus set size is discussed in section 2.5). However, despite using a large (nearly trial-unique) stimulus set, many thousands of trials were required for monkeys to reach criterion performance, after which they showed no evidence of forming long-term memories: their forgetting thresholds were between 30–50 s (Fritz et al., 2005). This was consistent with studies of auditory list memory that found monkeys could retain only about two sounds for delays up to 30s, the longest delays tested (Wright, 2007). Curiously, selective ablation of the rhinal cortex had no effect on auditory memory, and the auditory forgetting curves of normal monkeys resembled the visual forgetting curves of monkeys with bilateral ablations of the rhinal cortex (Fritz et al., 2005), suggesting that monkeys are natural born ‘auditory amnesiacs’.

The claim that monkeys do not possess long-term auditory memory is seemingly at odds with the ethological evidence for long-term auditory stores in nonhuman primates. Monkeys communicate through a repertoire of vocalizations that appear to carry representational meaning (e.g., food availability or the presence of a predator), and recognize conspecific vocalizations with respect to the kinship and social rank of the caller (Gil-da-Costa et al., 2003; Gouzoules et al., 1984; Rendall et al., 1996; Seyfarth et al., 1980). The auditory abilities of nonhuman primates could be mediated by at least two other types of memory that require a form of long-term storage, but are not sufficient to enable DMS. First, stimulus-response (or ‘habit’) learning and perceptual learning have been demonstrated by classical and instrumental conditioning in many species, and are associated with plasticity in the auditory cortex (Weinberger, 2004; Weinberger, 2015) and neostriatum (Xiong et al., 2015). Habit learning in the monkey depends upon the neostriatum but not the MTL (Mishkin, 1982). A second alternative is cross-modal association memory, in which a sound activates an associated image that is stored, presumably, in the cortical visual system. For instance, the association of individual voices to faces has been shown to arise spontaneously in monkeys for familiar conspecifics (Adachi and Hampton, 2011; Habbershon et al., 2013) as well as familiar humans (Sliwa et al., 2011), and vocalizations are associated with the facial expressions corresponding to their production (Ghazanfar and Logothetis, 2003; Payne and Bachevalier, 2013; Romanski, 2012). Monkeys can also learn arbitrary sound-picture associations by instrumental conditioning, in which a sound cues the monkey to select the associated image after a delay (Colombo and Gross, 1994; Fuster et al., 2000; Gaffan and Harrison, 1991); the reverse condition, in which an image cues the recollection of an associated sound, has not been demonstrated to our knowledge. That the sound stimulus activates an associated visual memory is supported by the finding that matching to the subsequent image was impaired by the presentation of visual – but not auditory – distractors during the delay (Colombo and Graziano, 1994).

The inability of monkeys to perform DMS beyond a few tens of seconds suggests that although some form of long-term trace exists in the brain (to support habit or associative learning), monkeys can not reactivate that trace for use in an identity comparison. A possible anatomical correlate of this inability lies in the paucity of auditory inputs to the MTL, which receives sensory input predominantly from higher visual cortex (Munoz-Lopez et al., 2010; Munoz-Lopez et al., 2015; Suzuki and Amaral, 1994). As the MTL is necessary for the formation of long-term visual memories (Mishkin, 1978), purely auditory stimuli may not have direct access to this mechanism of storage. This peculiarity of long-term auditory memory places important constraints upon models of short-term auditory memory.

2.2 Do monkeys possess auditory working memory?

Working memory (WM) refers to the ability to temporarily maintain and manipulate information. The term WM is sometimes used interchangeably with STM, and definitions of the two vary (Baddeley, 2007; Cowan, 2008), but WM is generally held to be an active process requiring attentional control and manipulation. By contrast, the simple storage of sensory information that is no longer present in the external world falls under the purview of STM, which we distinguish as passive STM (pSTM). Discrimination of two stimuli that differ along a single parameter (e.g., tone frequency, image contrast, or vibratory flutter frequency) across a brief delay is often referred to as a WM task (Pasternak and Greenlee, 2005; Romo and Salinas, 2003), but would fall under our definition of pSTM.

Working memory is believed to operate not on the maintained representation of the recently presented stimulus, but on the activation of a representation of that stimulus previously stored in LTM (Cowan, 1988; Cowan, 2008; Fuster, 1995; Jacob and Nieder, 2014; Norman, 1968). This would confer several advantages over pSTM. For one, stimuli could be classified into categories based on low-level or abstract properties, and that category could be held in memory rather than the full representation of the stimulus itself. This seems to be a critical factor in the capacity of visual memory, which is reduced from roughly 4–7 items to one or two items if those images can not be categorized (Olsson and Poom, 2005). Interestingly, when non-categorical auditory and visual stimuli are carefully equated in discriminability, the capacities of visual and auditory STM are equally limited (Visscher et al., 2007; but see Bigelow and Poremba, 2014). A second advantage of WM over pSTM is that the representation of an actively maintained item can be continually reactivated (or 'rehearsed'), which confers resistance to decay and to interference from other sensory stimuli encountered during the delay (Lewandowsky et al., 2009; Saults and Cowan, 2007).

2.3 Properties of auditory STM in monkeys: active WM or pSTM?

Without evidence of long-term recognition memory, the ability of monkeys to perform auditory DMS at short delays (5–30 s) was hypothesized to depend upon WM or STM (Fritz et al., 2005). This capacity was explored in later experiments using a serial DMS task with short (~1 s) delays (Scott et al., 2012; Fig 1), an auditory analog of an established visual paradigm (Eskandar et al., 1992; Miller et al., 1993; Miller et al., 1996). By the definition of WM above (section 2.2), the apparent absence of LTM would preclude the use of active WM, because there is no LTM trace to reactivate for on-line maintenance. If monkeys are instead dependent upon pSTM, this would lead to two predictions about their performance of serial DMS: (i) matching accuracy should be compromised by interference from nonmatch stimuli presented during the delay, and (ii) sound category should not play a role in matching behavior because categories themselves require LTM (Olsson and Poom, 2005).

Figure 1.

Timing and trial types for two versions of the auditory delayed-match-to-sample (DMS) paradigm. A correct trial is depicted for each trial type. In the task used by the Poremba group, a sample sound was followed by a test sound after a 5-s delay; after a mandatory wait period, the monkey pressed a button (and received a food reward) if the test sound matched the sample, or withheld response for a nonmatching test stimulus. Below, the serial DMS paradigm, in which sounds were separated by a 1-s delay and the matching test sound may appear after zero, one, or two intervening nonmatches. Release of a touch-bar following the match earned a liquid reward. Note that a match/nonmatch decision was required for the second and (on some trials) third stimulus in the sequence.

Monkeys’ performance of serial DMS confirmed both predictions, suggesting that auditory memory was reliant on a passively retained sensory trace (Scott et al., 2012). Accuracy was strongly dependent on trial type, such that performance was ~90% correct with no nonmatch stimulus intervening between sample and match but dropped to ~40% with two intervening nonmatch stimuli. Varying the interstimulus interval confirmed that performance was affected by the intervening stimuli, rather than by temporal decay. Active WM should be resistant to these nonmatch stimuli, but the strong effect of interference favors the interpretation that the monkeys used pSTM.

The brief (~300 ms) sounds used in this task were drawn from seven categories, including synthetic and natural sounds. In an attempt to make the task easier, the nonmatch sound(s) on a given trial were not drawn from the same category as the sample; e.g., if the sample was a pure tone, the nonmatching test sound could be a vocalization or modulated noise, but not another pure tone. Thus, while 'identity' was the intended rule for DMS performance, matching to category would have been sufficient. There was no evidence that monkeys extracted these categories in order to exploit this strategy. Surprisingly, their performance was slightly better for synthetic than for natural sound categories, including macaque vocalizations (Scott et al., 2012), which may be a general feature of auditory DMS with small stimulus sets (21 in this case; see also Gaffan and Harrison, 1991; Ng, 2011; effect of stimulus set size is discussed below in Section 2.5). Quantifying the spectral and temporal complexity of the stimuli confirmed that performance was better for sounds that were simpler in either dimension, particularly tones, noise, and frequency sweeps (Scott et al., 2013). From these data it appears that monkeys solved auditory DMS using a sensory or feature-based memory trace, which favors the retention of acoustically simple sounds, rather than a categorical or object-based representation.

2.4 Role of interference and acoustic features in pSTM

Whereas human subjects can be questioned as to the strategy they employed in solving a task, the strategy used by nonhuman subjects at best can be inferred from their behavior. The monkeys’ mediocre performance in the serial DMS task afforded the opportunity to infer, post-hoc, what criteria contributed to identifying a ‘match’ (Scott et al., 2013). In both monkey subjects, a select subset of sample/nonmatch pairs was particularly likely to elicit ‘false alarm’ (incorrect match) responses. Tellingly, this pattern was much stronger for the first nonmatch in the sequence, when the false alarm rate was relatively low, than for the second nonmatch, when the false alarm rate was higher and errors seemed to occur randomly among sample/nonmatch pairs. This would be predicted if the mnemonic trace forming a basis for the match comparison were degraded by the presentation of the intervening nonmatch. This strong effect of interference is more characteristic of sensory memory (pSTM) than of an actively maintained WM.

The pairs of stimuli eliciting high error rates seemed to share common spectral characteristics; for example, pure tones and band-pass noise of equal center frequency were often confused, despite their obvious qualitative difference to human ears (Scott et al., 2013). The error rate for a given sample/nonmatch pair was predicted by the spectral similarity (correlation of frequency spectra) between sounds; temporal similarity (between amplitude envelopes) was a much less effective predictor. The predominance of spectral cues in DMS behavior was confirmed by selectively removing spectral or temporal information from the original memoranda: performance with only temporal cues was at or near chance, whereas performance with only spectral cues was only slightly reduced from that for intact stimuli (Scott et al., 2013). The particular spectral feature that best predicted error rate was the spectral centroid, which correlates closely with "pitch" as estimated by analytical software (Boersma and Weenink, 2012). Rhesus monkeys may perceive pitch much as humans do (Joly et al., 2014), lending credibility to the idea that monkeys performed auditory DMS by relying primarily on a pitch cue.

Parallels between DMS performance in monkeys and pitch memory in humans suggest that the same pSTM mechanism underlies both faculties. Pitch is a continuous sound quality that is largely immune to verbal labels or categorization in most listeners. Pitch memory begins to decay at intervals ≥3 s (Harris, 1952), consistent with the temporal limits of DMS performance in macaques (Fritz et al., 2005; Scott et al., 2012). Pitch memory is independent of verbal memory, in that it is not affected by interference from verbal material presented during the delay (Deutsch, 1970) but is nevertheless highly susceptible to interference from tones close in frequency to the sample (Deutsch, 1972; Zatorre et al., 1994). Similar manipulations of interference indicate that attributes like pitch, loudness, and timbre are represented independently in STM (Caclin et al., 2006; Clement et al., 1999; Nousak et al., 1996; Semal and Demany, 1991). This feature-based representation distinguishes pSTM from WM, which operates on configural or item-based representations (Joseph et al., 2015).

2.5 Cross-trial interference in auditory and visual STM

We have proposed that an advantage of WM over pSTM is resistance to interference. Thus far we have referred to retroactive interference within a trial, in which a nonmatch stimulus interferes with the previously presented sample being held in memory (referred to as ‘inhibition’ by (Wright, 2007). A related process is proactive interference, in which a previously presented sound affects the maintenance of a subsequently presented sample. Susceptibility to proactive interference offers a glimpse into the strategy that an animal may be employing to solve a DMS task.

If the stimulus set is large, then items will seldom re-appear from trial to trial, and successful performance could be achieved merely by judging if an item had been presented recently. In this case, a passive memory mechanism would suffice to identify the match. But if the stimulus set is small, then the match stimulus from a given trial may reappear as the nonmatch on a subsequent trial. A recency criterion, perhaps supported by pSTM, would lead to false-alarm errors in response to nonmatch stimuli that had appeared in a previous trial. By contrast, an active WM process would allow for the memory buffer to be ‘reset’ between trials (Yakovlev et al., 2013), and for the sample on the current trial to be selectively attended and maintained. Under these conditions, cross-trial proactive interference would be minimal.

The effect of proactive interference on auditory memory in the monkey was addressed by Wright using a list memory paradigm (Wright, 1999; Wright, 2007), and by Poremba and colleagues using a non-serial DMS paradigm (Bigelow and Poremba, 2013a; Bigelow and Poremba, 2013b; Bigelow and Poremba, 2015). Consistent with the attentional demand induced by repeating stimuli between trials, accuracy decreased with stimulus set size (Bigelow and Poremba, 2013b; Wright, 1999; Wright, 2007). Effects were detectable for stimuli that had appeared several trials back, spanning up to 3 minutes, suggesting that the effects of interference endured beyond the memory span that could be inferred from the current performance of the DMS task alone (Bigelow and Poremba, 2013a). However, we would argue that proactive interference evident in our data and others’ reflects a stimulus-response association that is distinct from the recognition memory that DMS is designed to test (Scott et al., 2012); if the response to a stimulus on the previous trial was rewarded, that reinforcement increases the probability of response when the same stimulus is presented as a nonmatch in the current trial. Overall, the effects of proactive interference are consistent with the idea that monkeys rely on a passive short-term sensory store to perform auditory STM tasks. Recent evidence suggests that this may not be unique to audition.

It has been argued that the monkey’s cortical memory mechanisms in audition are intrinsically different from those in vision and touch (Fritz et al., 2005), and that this difference in LTM extends to their reliance on pSTM rather than active WM. This distinction between modalities rests on prior evidence from monkeys performing visual DMS, in which active WM is used to selectively memorize a sample image while interfering distractor images are ignored or forgotten (Basile and Hampton, 2013; Miller et al., 1993; Miller and Desimone, 1994; Yakovlev et al., 2013). However, more recent evidence challenges the assumption that this necessarily reflects true WM, suggesting instead that monkeys rely on a passive recency criterion to solve serial DMS in the visual domain (Wittig and Richmond, 2014). Even when required to memorize only the first item in a sequence, residual effects of proactive interference reveal that monkeys will not use selective memory exclusively (Wittig and Richmond, 2014). Our data align with those of Wittig and Richmond to suggest that passive memory for stimulus recency is the default mechanism for STM in nonhuman primates across both audition and vision.

2.6 Differences across monkeys, apes, and humans

What endows humans with our ability to store and recognize words, songs, and other sounds? One theory is that humans enlist the inferior frontal motor areas underlying speech production, which could transform a fluctuating acoustic stimulus into a subvocal oromotor sequence (Galantucci et al., 2006; Hickok and Poeppel, 2007; Liberman and Mattingly, 1985; Rauschecker and Scott, 2009). This motor theory of auditory memory is supported by the finding that recognition memory for reversed words, which cannot be pronounced, is inferior to that for pronounceable pseudo-words (Schulze et al., 2012). This transformation is likely to involve Broca’s area (Flinker et al., 2015) and thus to depend upon the arcuate fasciculus, the pathway connecting posterior temporal auditory regions to the inferior frontal cortex (Catani and Mesulam, 2008; Petrides and Pandya, 2009). Temporary disruption of the posterior auditory cortex by transcranial magnetic stimulation impairs recognition (but not discrimination) of pseudowords, implicating the caudal terminus of the arcuate fasciculus in the storage of speech sounds (Karabanov et al., 2015).

The arcuate fasciculus is prominent in humans, of intermediate strength in chimpanzees, and very weak in macaques (Rilling et al., 2008). Despite their stronger arcuate fasciculus, chimpanzees perform no better than macaques on tests of auditory memory (Hashiya and Kojima, 2001; Siebert et al., 2013), suggesting that the mnemonic advantage conferred by the human dorsal-stream language pathway is all-or-none. The extent to which the dorsal auditory-motor pathway in humans is specialized relative to that in nonhuman primates is a topic of debate (Bornkessel-Schlesewsky et al., 2015a), but to compare auditory memory systems between species any linguistic advantage (e.g., labeling or mimicry) must be carefully circumvented to study sensory STM in isolation from verbal memory.

3. Cortical systems for auditory STM

Where in the brain should we expect to find the neural signature of STM? Contemporary theories favor the view that memory is supported by the same systems that enable perception (i.e., sensory and association cortex) as opposed to a redundant memory-specific buffer elsewhere in the brain (Cowan, 2008; Postle, 2006). This would focus the search for neural correlates of pSTM in the monkey on the auditory cortex, from the core areas (primary recipients of thalamic input) to the belt, parabelt, and higher-order regions comprising the ventral sound processing stream (Hackett, 2011; Kikuchi et al., 2010; Munoz-Lopez et al., 2010; Rauschecker and Scott, 2009). Beginning with the secondary belt regions, every stage of the auditory cortical hierarchy is interconnected with the prefrontal cortex (PFC; Plakke and Romanski, 2014; Romanski et al., 1999a; Romanski et al., 1999b), which is known to support visual STM through its interactions with higher-order visual cortex (Fuster, 1973; Fuster et al., 1985; Miller et al., 1996). This section will review an emerging body of work that identifies a similar temporo-frontal network that may support auditory STM in the nonhuman primate.

3.1 Neural correlates of pSTM

Studies of single-unit activity during DMS performance have identified two general classes of effects potentially associated with STM. First, the sensory response (i.e., firing rate) of a neuron to a given stimulus may be modulated by the context in which that stimulus appears (sample, match, or nonmatch). This modulation is typically referred to as match suppression (MS) or match enhancement (ME; Miller and Desimone, 1994), though as discussed below, effects of this type (particularly suppression; section 3.1.2) can be observed outside of an explicit memory task. Second, activity may be modulated (elevated or suppressed) during the delay period between the sample and test stimuli. These phenomena were initially identified in inferotemporal (IT) visual cortex during visual STM (Baylis and Rolls, 1987; Brown et al., 1987; Fuster et al., 1981; Fuster and Jervey, 1982; Miyashita and Chang, 1988). Contemporaneous experiments in the auditory domain reported similar effects in the secondary auditory cortex of a baboon comparing tone frequencies across a 1-s delay (Gottlieb et al., 1989). Half of the neurons recorded exhibited activity during the delay that was dependent on the frequency of the sample tone, and a quarter exhibited a match/nonmatch difference.

Recent studies have re-opened the question of how cortical neurons represent auditory STM in the nonhuman primate model (Fig 2; Bigelow et al., 2014; Ng et al., 2014; Plakke et al., 2013; Scott et al., 2014). As described above (section 2.2), Scott et al. (2012; 2012; 2014) applied a serial DMS paradigm using stimuli and delays of a similar duration to those of Gottlieb et al. (1989) but with two key differences: stimuli comprised a range of acoustic categories, and intervening nonmatch stimuli could appear between the sample and match (Fig. 1). This created a direct auditory analog to a class of serial visual DMS tasks used in studies of IT visual cortex (Eskandar et al., 1992; Miller et al., 1993). Neurons were recorded from areas referred to collectively as the rostral superior temporal cortex (rSTC), including rostral core, belt, parabelt, and higher-order areas (Scott et al., 2014). Part of the recorded region was co-extensive with the rostral superior temporal gyrus lesion applied by Fritz et al. (2005; see section 2.1), which reduced forgetting thresholds from 30–50 s to <10 s, implicating this region in supporting auditory DMS. A complementary set of studies from Poremba and colleagues recorded from areas bracketing the rSTC at lower and higher stages of cortical processing: the primary auditory cortex (AI; Bigelow et al., 2014), and the dorsal temporal pole (dTP: Ng et al., 2014), as well as the dorsolateral PFC (Plakke et al., 2013). The following sections (3.1–3.3) integrate these findings to outline a temporal-frontal network engaged during auditory STM in the nonhuman primate.

Figure 2.

Cortical regions recorded in neurophysiological studies of auditory STM in monkeys, collected on a schematic depiction of the macaque brain. Top, the bold black line highlights the lateral sulcus (ls), which has been opened up to reveal the auditory areas of the supratemporal plane (STP) illustrated below. The ascending hierarchical levels of core, belt, and parabelt are shaded from dark to light gray. Areas recorded in frequently cited studies are indicated in bold type and outlined in blue: the primary auditory cortex (AI, a subdivision of the core), caudal belt, rostral superior temporal cortex (rSTC), and dorsal temporal pole (dTP). Note that rSTC includes rostral core and belt, as well as the superior temporal gyrus (STGr), rostral STP, and some overlap with the dTP. The dorsolateral (dlPFC) and ventrolateral (vlPFC) prefrontal cortex is indicated on the lateral view at the top. A right-hemisphere view is used for convenience, but in actuality recordings were obtained from the right hemisphere in caudal belt (N=1 baboon; Gottlieb et al., 1989), one right and three left hemispheres in rSTC (N=3 rhesus monkeys; Scott et al., 2014), one left and two right hemispheres in vlPFC (N=2 rhesus, Hwang and Romanski, 2015; and the left hemisphere in AI, dTP, and PFC (N=2 rhesus; Bigelow et al., 2014; Ng et al., 2014; Plakke et al., 2013).

3.1.1 Delay-period activity

Spiking activity during the inter-stimulus delay period has been consistently identified across studies of auditory DMS. Activity during the delay period was shifted from the baseline rate in about one third of rSTC neurons, with an equal split between delay enhancement (elevation of spike rate) and delay suppression (Scott et al., 2014). Similar responses were seen in the adjacent dTP region and in AI: among the 20–25% of units that showed an effect, a roughly equal proportion showed suppression or enhancement (Bigelow et al., 2014; Ng et al., 2014). Delay enhancement and suppression in rSTC showed different dynamics across the trial: enhancement and suppression were equally common in the first delay period, but in the second delay period (after the presentation of a nonmatching stimulus) enhancement was diminished; by contrast, suppression was robust across two intervening nonmatch stimuli (Scott et al., 2014). The decrement in the proportion of units showing delay excitation coincided with a drop in behavioral performance (matching was much more accurate for the first test stimulus than for the second test stimulus, which followed an intervening nonmatch). This is suggestive of a role for delay enhancement in maintaining the sensory trace upon which DMS performance depends, but clarifying the role of this activity will require further study.

Delay activity could be conceived as a continuation of the sensory response evoked by the sample, bridging the temporal gap between the sample and test stimuli. However, two aspects of the delay activity observed during auditory DMS are incompatible with this idea. First, delay activity in rSTC was only weakly selective for the preceding sample, even among the subset of neurons exhibiting delay enhancement (Scott et. al, 2014). This contrasts with the sample-selective delay activity reported previously (Gottlieb et al., 1989). (A possible reason for the discrepancy is that the earlier study used a small set of tones selected to elicit strong responses from tonotopically-organized cortex, whereas the rSTC recordings were made with a larger set of complex sounds that was not tailored to each neuron.) Second, delay activity in AI and dTP was not stable across the full 5-s delay used in those studies (Bigelow et al., 2014; Ng et al., 2014); this instability could not be observed in studies that used a shorter 1-s delay (Gottlieb et al., 1989; Scott et al., 2014).

It is generally believed that selective delay activity in IT visual cortex is not the basis for serial DMS performance, but the reasons behind that claim are difficult to interpret in the case of auditory DMS. For example, delay activity in IT cortex does not survive an intervening nonmatch stimulus, yet matching behavior remains accurate (Miller et al., 1993), which rules out delay activity as the relevant signal (Sugase-Miyamoto et al., 2008). In the auditory domain, delay enhancement and DMS performance drop off in tandem as intervening stimuli appear in the trial (Scott et al., 2014), so the same argument does not apply. In addition, delay activity in visual cortex was reported only for familiar stimuli (those presumably retrieved from LTM), yet monkeys were capable of performing DMS for novel visual stimuli (Miyashita and Chang, 1988). Similarly, disruption of delay activity in PFC by microstimulation only affected visual DMS for familiar stimuli, linking this activity to recall of a stored memory rather than to sensory maintenance (Sobotka and Ringo, 1993). If monkeys are unable to temporarily reactivate representations of sounds from LTM for use in WM (see sections 2.1 and 2.2, above), as humans appear to do with visual images (Lewis-Peacock and Postle, 2008), then delay activity in auditory DMS may represent a different process.

Some of the evidence above suggests that delay activity, despite its ubiquity, does not carry the sustained sensory signal that we propose is the basis for auditory DMS performance. However, the relatively recent application of multivariate analysis methods to fMRI data has shown that information about a stimulus in STM can be decoded from the activity of sensory cortex, both visual (Harrison and Tong, 2009; Serences et al., 2009) and auditory (Linke et al., 2011; Linke and Cusack, 2015); for a recent review, see (Sreenivasan et al., 2014). Before the role of delay activity is dismissed, similar decoding methods should be applied to spiking activity from macaques during auditory STM to determine whether a representation exists at the population level that is not evident in single neurons (Machens et al., 2010; Mante et al., 2013; Meyers et al., 2008; Rigotti et al., 2013).

3.1.2 Match response modulation

In the absence of persistent spiking activity in single units, a memory trace could lie in a subthreshold shift in synaptic weights (Sugase-Miyamoto et al., 2008), which may manifest as match suppression (see above, section 3.1). Averaged over the population of neurons, responses in all three cortical regions (AI, rSTC, and dTP) were diminished for match stimuli relative to the same stimuli presented as nonmatch or sample (Bigelow et al., 2014; Ng et al., 2014; Scott et al., 2014). In all three areas, a transient MS emerged as early as 30–50 ms from stimulus onset. The proportion of individual units showing MS varied across areas: 23% in AI vs. 9% in dTP across a 5-s delay, and 25% in caudal belt vs. 12 % in rSTC across a 1-s delay (Bigelow et al., 2014; Ng et al., 2014; Gottlieb et al., 1989; Scott et al., 2014). However, these proportions may not be directly comparable due either to differences in statistical criteria across studies, and/or to different firing rates between neurons in primary and higher-level cortex.

The most parsimonious explanation of MS is response adaptation, a ubiquitous feature of sensory neurons that is present without any explicit memory task, and even under anesthesia (Perez-Gonzalez and Malmierca, 2014). Adaptation effects lasting 1 s are commonplace in AI (Bartlett and Wang, 2005; Brosch and Scheich, 2008), and effects lasting up to 5 s have been reported (Werner-Reiss et al., 2006), a period long enough to span the interval in the Bigelow et al. (2014) task. The time scale of adaptation in higher-order auditory areas (rSTC and dTP) has not been systematically studied, but is likely to be longer than that at more caudal stations in the ventral pathway (Bendor and Wang, 2008; Lu et al., 1992; Scott et al., 2011). However, MS during serial DMS was observed even with one or two intervening nonmatch stimuli (Scott et al., 2014), suggesting that rSTC neurons may show a particularly robust form of adaptation that is inherent to these areas, or recruited during DMS performance. This latter idea, that some form of attentional control may enhance inherent adaptive processes, is supported by the observation that MS (and delay activity) were observed to a greater degree during STM behavior than during passive listening (Gottlieb et al., 1989). Until MS is demonstrated to be selectively engaged during a STM task, the most conservative stance is to follow Miller and Desimone (1994) and consider MS a passive mechanism that is intrinsic to higher stages of cortical sensory processing.

Although stimulus repetition caused suppression of the population response, a small subset of neurons in rSTC (7%) exhibited the opposite effect, match enhancement (Scott et al., 2014). Whereas MS appeared at the same latency as the sensory response itself, ME lagged MS by >50 ms. Unlike MS, ME is not easily explained as a form of adaptation. And much like the difference between excitatory and suppressive delay-period effects, ME and MS displayed different dynamics across the trial: the prevalence of ME was significantly reduced after an intervening nonmatch, but MS was robust across all trial lengths (Scott et al., 2014). Thus the dynamics of ME matched that of the monkeys’ behavioral accuracy, which declined after an intervening stimulus. Match enhancement arose considerably later in area AI (~250 ms after test stimulus onset) than in rSTC (~100 ms), and was evident only in the multi-unit response. (The single-unit population in AI exhibited ME only around the time of the behavioral response, which may result from top-down feedback related to response selection [Bigelow et al., 2014; Brosch et al., 2005] but not the match per se.) The relative timing of the ME signal across areas suggests that the relatively early ME in the rSTC originates locally and propagates back to AI via recurrent connections; alternatively, ME originating in PFC (see section 3.3, below) may propagate back to AI via the higher-order fields of the rSTC. Simultaneous recordings from the temporal and prefrontal cortex could clarify the dynamics of this network during auditory STM.

In IT visual cortex, robust ME was observed only under a particularly demanding DMS task in which nonmatch stimuli were repeated within the trial (e.g., ‘ABBA’), requiring the monkey to detect the repetition of the first item exclusively, not any recently presented stimulus (Miller and Desimone, 1994). Because ME was seen only for the match, but not the repeated nonmatch, Miller and Desimone (1994) associate ME with ‘active’ memory (selective memorization; but see Wittig and Richmond, 2014). By contrast, MS was also observed for the repeated nonmatch and was interpreted as a passive mechanism signaling the recency of stimulus appearance irrespective of task demands (Miller and Desimone, 1994). To our knowledge monkeys have never been trained on this variant of auditory DMS, but from the available evidence an ‘active’ WM robust to intervening distractors (and its associated ME) is weak or absent in the auditory domain. Studies of auditory DMS in humans should use a comparable serial paradigm, including distractor stimuli, to determine whether match enhancement and delay activity in the human cortex is robust to interference, and capable of supporting active auditory WM.

3.2 Neural effects of stimulus repetition

The enhancement and suppression of match responses in single neurons during DMS may be related to more general effects of stimulus repetition that are evident in population measures of neural activity. In studies using functional MRI, a reduction in response magnitude to a repeated stimulus is typically called repetition suppression, and is thought to reflect a correlate of priming that may underlie other memory processes as well (Grill-Spector et al., 2006). Importantly, repetition suppression is stimulus-specific, and that specificity has been used to infer the stimulus selectivity of the underlying neuron population (Krekelberg et al., 2006; but see Sawamura et al., 2006). Several neural models could plausibly explain repetition suppression (Grill-Spector et al., 2006), including a population-wide decrease in activation (fatigue), a selective decrease among a subset of neurons that results in a sparser representation (sharpening), or a decrease in the latency and duration of activation that does not necessarily affect the response magnitude (facilitation). These models are not mutually exclusive, but aspects of MS in single neurons of auditory cortex are most compatible with the sharpening model: only a subset of neurons exhibit MS (not all, as the fatigue model would predict), and match responses have not been shown to have shorter latencies (as the facilitation model would predict).

Less widely discussed than repetition suppression is the complementary effect of repetition enhancement of the fMRI signal. Whereas repetition suppression may be explained by adaptation, repetition enhancement has been associated with several cognitive variables including stimulus recognition, learning, attention, and expectation (Segaert et al., 2013). Auditory-verbal STM elicits repetition suppression localized to the rostral STG, as well as repetition enhancement in caudal STG (Buchsbaum and D'Esposito, 2009). Suppression in rostral areas is consistent with single-unit data from macaques (Ng et al., 2014) and points to a common ventral-stream mechanism for sensory STM. Match enhancement was also observed in single units in rostral STG (Scott et al., 2014) but was less common than suppression and may not be evident in a spatially averaged fMRI signal. Repetition suppression and enhancement can be detected concurrently in extrastriate visual cortex using multivariate analysis methods (de Gardelle et al., 2013), which may reveal a similarly mixed response in the human auditory cortex. Whether the enhancement effects indicated by fMRI (Buchsbaum and D'Esposito, 2009) are evident among single units in caudal STG is unknown, which underscores the need for further studies of the dorsal auditory pathway in nonhuman primates.

The spatial resolution of fMRI allows for the cortical locus of suppression and enhancement to be distinguished (Buchsbaum and D'Esposito, 2009), but the temporal resolution of fMRI is coarse. The reverse is true of electro-and magneto-encephalography (EEG and MEG), which can identify differences in timing between enhancement and suppression that may parallel those observed in single units. Evidence from MEG identifies both repetition suppression and repetition enhancement in the establishment of auditory memory traces in humans (Recasens et al., 2015; Rong et al., 2011); like the MS and ME observed in single units during DMS tasks in monkeys, these effects show a difference in timing: initial suppression, followed by later enhancement (Recasens et al., 2015).

The effect of repetition in auditory sequences is most often studied using an oddball paradigm, in which the response to a given sound is dependent upon its appearance as a ‘standard’ within a repeating sequence (which induces adaptation), or as an infrequent ‘deviant’ within that sequence. These repetition effects, namely the ‘mismatch negativity’ (MMN, the difference between the evoked potential under the deviant and standard condition) are interpreted as deviance detection, i.e. a form of sensory memory or predictive coding within the auditory system (Baldeweg, 2006; Friston, 2005; Naatanen and Winkler, 1999). However, a strong case has been made that the MMN can be explained through simple adaptation in the auditory cortical network (May and Tiitinen, 2010). This would position the MMN as the EEG/MEG correlate of the stimulus-specific adaptation that is evident in single units of the anesthetized auditory cortex under similar oddball paradigms (Nelken and Ulanovsky, 2007; Ulanovsky et al., 2003; Ulanovsky et al., 2004). Stimulus-specific adaptation has been demonstrated in AI of the awake macaque (Fishman and Steinschneider, 2012), and may be related to the match suppression that we propose to underlie passive short-term memory in the nonhuman primate. Although the oddball paradigm is quite different from the DMS tasks that are the focus of this review, the same mechanism of pSTM may support DMS as well as other auditory memory tasks (Brosch et al., 2004; Hwang and Romanski, 2015; Russ et al., 2008).

3.3 Is auditory STM distributed or localized?

If STM maintenance is supported by the same areas that support perception, then a pitch memory task should engage pitch processing centers (Grimault et al., 2014). Pitch sensitivity in human auditory cortex appears to follow a caudorostral gradient (Norman-Haignere et al., 2013), which would predict some degree of specialization for pitch memory in the fields rostral to AI, for which the rSTC in the monkey may be a homolog. But in general the representation of pitch, timbre, and other auditory features appears to be widely distributed in the auditory cortex (Bizley et al., 2009), which would predict a similarly distributed memory trace if subjects were relying on a pitch-based cue as monkeys’ behavior suggests (section 2.4, above). Accordingly, neural effects related to STM have been observed across nearly the entire supratemporal plane, from the caudal belt (Gottlieb et al., 1989) to the dTP (Ng et al., 2014).

Neural correlates of auditory STM have also been described in the same regions of PFC that are active during visual STM (Fig. 2). In dorsolateral PFC, the population response exhibits a sustained ME effect, appearing at a latency that is possibly short enough to drive ME in lower cortical regions by top-down modulation (Plakke et al., 2013). Neurons in ventrolateral PFC exhibit match and delay effects during a multimodal (face/voice) memory task (Hwang and Romanski, 2015), and inactivation of ventrolateral PFC impairs auditory and auditory-visual memory performance (Plakke et al., 2015). Although ventrolateral PFC is strongly interconnected with the ventral stream of auditory cortex (i.e., the rSTC and dTP; Romanski et al., 1999a; Romanski et al., 1999b), neurons in ventrolateral PFC were predominantly sensitive to changes the visual component of the memoranda (Hwang and Romanski, 2015). This identifies ventrolateral PFC as a site of multimodal integration, rather than a a specialized module for auditory STM. Taken together, the available data support a distributed temporo-frontal network that subserves auditory STM.

An advantage of a distributed representation would be resilience to cortical damage, as was observed by Fritz et al. (2005): removal of the rostral STG reduced the monkeys’ forgetting threshold, but did not affect performance at the shortest delays tested (5 s). The hierarchical organization of the auditory system may serve to build a lengthened sensitivity to stimulus history (Hasson et al., 2015), which would manifest as MS or ME across progressively longer time scales at higher stages of processing. Within each stage, the differential response of populations of neurons that do or do not display MS (or ME) would signal the relative recency of stimulus occurrence, as has been proposed to operate in IT visual cortex (Miller et al., 1993).

4. Conclusions

The argument that auditory LTM is absent in the monkey (Fritz et al., 2005), and that STM is limited relative to that in vision (Scott et al., 2012), appears to undermine a key rationale for studying hearing in the nonhuman primate: viz., that the behavior, neuroanatomy, and perceptual ability of monkeys closely resemble our own (Belmonte et al., 2015; Seyfarth and Cheney, 2014; Stebbins and Moody, 1994). However, several lines of evidence argue that studies in nonhuman primates (and animal models in general) are applicable to the human auditory system, including auditory memory.

First, if retention of a pitch trace is independent of vocal processes, then the pSTM evident in the nonhuman primate offers a tenable model of the neural mechanisms supporting this component of auditory memory. Sensitivity to stimulus history is intrinsic to auditory perception and appears to occur automatically (Demany and Semal, 2007). Pitch memory is largely divorced from verbal and oromotor processes, as demonstrated by the fact that memory is not improved by allowing subjects to hum the remembered pitch, and that articulatory suppression does not interfere with tone retention (Demany and Semal, 2007; Deutsch, 1970; Guimond et al., 2011). Although active rehearsal bolsters verbal STM, it is largely irrelevant to auditory sensory STM.

Second, neuroimaging studies have identified similarities between the cortical systems supporting auditory STM in human and in nonhuman primates (see section 3.2, above). Suppression of responses to match stimuli was observed in rostral STG of human subjects (Buchsbaum and D'Esposito, 2009) as well as in single units of monkeys (Ng et al., 2014; Scott et al., 2014). Suppression occurs at a shorter latency than enhancement in MEG recordings from humans, and in single neuron recordings from monkeys (Recasens et al., 2015; Scott et al., 2014). These parallels suggest at least some overlap in pSTM mechanisms between the auditory ventral pathway in humans and monkeys.

Third, the disparity between visual and auditory memory may not be unique to monkeys. Recent evidence shows that human auditory memory is inferior to visual memory, whether for colors and tones in STM or complex objects in LTM (Bigelow and Poremba, 2014; see also Cohen et al., 2009). Connectional neuroanatomy in the monkey reveals an impoverished pathway from the auditory cortex to the medial temporal lobe, which may underlie the relative difficulty of forming and recalling auditory memories relative to visual memories (Munoz-Lopez et al., 2010; Munoz-Lopez et al., 2015). The homology between auditory memory pathways in the human and nonhuman primate is a critical open question, but both species may be operating under similar constraints.

The advantage for humans is likely to lie not in the ventral stream, but in the dorsal stream associated with mapping sounds to articulatory movements. Whether the dorsal stream in monkeys differs qualitatively or only quantitatively from that in humans remains a subject of debate (Bornkessel-Schlesewsky et al., 2015a; Bornkessel-Schlesewsky et al., 2015b; Skeide and Friederici, 2015). But the parallels in behavior, anatomy, and physiology between monkeys and humans suggest that the mechanism of pSTM in the nonhuman primate offers a tenable model of the ventral stream mechanisms supporting auditory sensory memory in humans (Kaiser, 2015), in isolation from related mechanisms of verbal memory.

Highlights.

• Mechanisms of auditory memory may differ between species and between modalities

• Auditory passive short-term memory (pSTM) is comparable between humans and monkeys

• Correlates of pSTM are evident in neurons throughout auditory and prefrontal cortex

• Auditory and visual STM may share some common neural mechanisms

Abbreviations

pSTM

passive (auditory) short-term memory

DMS

delayed match-to-sample

LTM

long-term memory

STM

short-term memory

DE

delay enhancement

DS

delay suppression

ME

match enhancement

MS

match suppression

MTL

medial temporal lobe

AI

primary auditory cortex

rSTC

rostral superior temporal cortex

dTP

dorsal temporal pole

PFC

prefrontal cortex

STGr

rostral superior temporal gyrus

EEG

electroencephalography

MEG

magnetoencephalography

MMN

mismatch negativity