Explicit Processing Demands Reveal Language Modality-Specific Organization of Working Memory

Abstract

The working memory model for Ease of Language Understanding (ELU) predicts that processing differences between language modalities emerge when cognitive demands are explicit. This prediction was tested in three working memory experiments with participants who were Deaf Signers (DS), Hearing Signers (HS), or Hearing Nonsigners (HN). Easily nameable pictures were used as stimuli to avoid confounds relating to sensory modality. Performance was largely similar for DS, HS, and HN, suggesting that previously identified intermodal differences may be due to differences in retention of sensory information. When explicit processing demands were high, differences emerged between DS and HN, suggesting that although working memory storage in both groups is sensitive to temporal organization, retrieval is not sensitive to temporal organization in DS. A general effect of semantic similarity was also found. These findings are discussed in relation to the ELU model.

Working memory is the limited cognitive capacity available for online processing and temporary storage of information and is thus crucial to language processing. Behavioral and neuroimaging data suggest that the architecture of working memory for sign language, which is the preferred language of the congenitally deaf, is largely similar to that for speech, although there are some modality-specific differences: Temporary storage in working memory is less capacious for signs than words (Boutla, Supalla, Newport, & Bavelier, 2004; Geraci, Gozzi, Papagno, & Cecchetto, 2008; Marschark & Mayer, 1998); the organization of working memory for sign language does not seem to support temporal information in the same way as working memory for speech (Wilson, Bettger, Niculae, & Klima, 1997), and working memory for sign language engages additional neural structures (Rönnberg, Rudner, & Ingvar, 2004; Rudner, Fransson, Ingvar, Nyberg, & Rönnberg, 2007). The nature of these differences and their interplay is still unclear. In this study, we systematically investigate working memory storage processes in Deaf Signers (DS), Hearing Signers (HS), and Hearing Nonsigners (HN) in a set of three experiments.

Baddeley's Model of Working Memory

Sensory differences form the basis for the verbal and nonverbal slave systems that are the key components of one influential model of working memory (Baddeley, 1986; Baddeley & Hitch, 1974). Apart from these systems, the phonological loop, and the visuospatial sketchpad, Baddeley's model includes a central executive, which controls the two slave systems, and a recently added episodic buffer (Baddeley, 2000; Repovs & Baddeley, 2006). Empirical evidence provides support for characterizing the phonological loop in terms of a passive temporary store and an active processing loop that has the dual function of reviving decaying representations in the store by means of subvocal repetition and recoding nonphonological sensory input into phonological representations. The active processing loop is what distinguishes working memory from store-based theories of memory (e.g., Atkinson & Shiffrin, 1968) in terms of which short-term memory is a passive store in which old items are displaced by new ones or transferred to long-term memory by means of rehearsal.

The phonological loop is the most widely investigated component of Baddeley's model (Baddeley, 2000), but empirical support for the other components is amassing. Logie (1995) argued that the visuospatial sketchpad can be divided into a passive store (visual cache) and an active process (inner scribe). More recent work supports fractionation of the visuospatial sketchpad into visual and spatial components (Klauer & Zhao, 2004). The central executive is concerned with functions such as switching plans, time-sharing in dual tasks, selective attention, and temporary activation of long-term memory (Baddeley, 1997). The episodic buffer has recently been fractionated from the central executive to meet the theoretical need for a component that serves the function of forming and maintaining unitary multidimensional representations in working memory (Repovs & Baddeley, 2006). These representations may be based on sensory information in different modalities or mnemonic information in various memory systems or a combination of both. Thus, the episodic buffer comprises a binding mechanism. Unlike the phonological loop and the visuospatial sketchpad that deal with verbal and visuospatial information, respectively, the episodic buffer deals with multimodal information.

Working Memory for Sign Language

It has been seen as a challenge to investigate Baddeley's model of working memory in relation to sign language. Signed languages are natural languages that exist in cultures the world over where deaf people meet (Emmorey, 2002). They fulfill linguistic requirements in terms of phonology, morphology, syntax, semantics, and prosody (Klima & Bellugi, 1979) and are the preferred languages of the congenitally deaf. Crucially, in terms of memory and cognition, signed language is verbal but at the same time visuospatial. Thus, it is not easily explained in terms of modality-specific models of cognition.

Wilson and Emmorey (1997, 1998, 2003) tested the explanatory power of Baddeley's model in relation to working memory for sign language in a series of experiments. Using sets of phonologically similar/dissimilar signs (same/different handshape) and long/short signs (long/short movement path), they found that working memory for sign language displayed equivalents of the classic effects of phonological similarity (Wilson & Emmorey, 1997) and word length (Wilson & Emmorey, 1998) associated with the phonological loop. On the basis of this, they argued that working memory can develop a language-based rehearsal loop in the visuospatial modality, thus providing support for a modality-neutral model of working memory, processing abstract representations in the same way, irrespective of input modality.

Modality-Specific Differences in Working Memory for Sign Language

In a further study (Wilson & Emmorey, 2003), they showed that DS were disrupted on recall of lists of ASL signs by irrelevant visuospatial material, whereas the same was not true of hearing speakers memorizing lists of words. This suggests that working memory representations of sign language, although they have modality-neutral characteristics, do have visuospatial characteristics, implying sensorimotor coding (Wilson, 2001) and modality specificity.

Further modality-specific differences in working memory for sign and speech concern capacity and organization of the temporary store. Sign span is generally around 5 ± 1 items (Boutla et al., 2004; Geraci et al., 2008) compared to digit span, the cardinal test of speech-based working memory, which is around 7 ± 2 items (Miller, 1956). The older literature on short-term memory function in the deaf consistently reported lower capacity for deaf people than for hearing people (e.g., Conrad, 1972). However, these studies were often based on processing of speech-based stimuli, which puts deaf people at an obvious disadvantage. More recently it has been shown that native HS display a lower span for sign-based than for speech-based stimuli (Boutla et al., 2004; Rönnberg et al., 2004), demonstrating that lower span effect for sign language is not just an artifact of deafness.

Explaining Lower Sign Span

Several explanations of the discrepancy between sign and speech capacity have been proposed. These include intermodality differences in (a) articulation rate, (b) phonological properties, (c) retention of auditory and visual information, and (d) retention of temporal order information.

Articulation rate.

In a study of the word-length effect, Baddeley, Thomson, and Buchanan (1975) found that span was equivalent to the number of words that could be read out in 2 s. Thus, they proposed that the capacity of the phonological loop is determined by rate of rehearsal of stored items. Although signed propositions are no lengthier than their spoken equivalents, individual signs take longer to articulate than individual words (Bellugi & Fischer, 1972), and Wilson and Emmorey (1998) found that this characteristic is retained in working memory representations, thus, presumably, affecting rate of rehearsal. However, there is evidence that lower sign span cannot be fully explained in terms of articulation rate. In a previous study, we found that immediate serial recall performance for sign and speech did not correlate with response time (Rönnberg et al., 2004), and Boutla et al. (2004) found a span discrepancy even when articulation rate of signs matched that of words. Thus, differences in articulation rate for signs and words cannot fully explain span discrepancies.

Phonological properties.

Boutla et al. (2004) investigated whether span differences for sign and speech remain when the phonological similarity of stimulus materials is held constant over modalities. For speech materials they used lists of digits from one to nine that are phonologically dissimilar. In sign language, digits display high phonologically similarity. Fingerspelled letters, on the other hand, display phonological diversity. Thus, for the sign materials, a set of fingerspelled letters in American Sign Language (ASL) was selected that was deemed to be similar to the set of spoken digits in its lack of mutual phonological similarity. Results showed that sign/speech span differences persisted despite the stimulus matching procedure, suggesting that phonological similarity among sign digits is not a sufficient explanation of intermodal span differences.

Emmorey and Wilson (2004) argued that digits have a special status in human cognitive processing and that the sign/speech span differences demonstrated by Boutla et al. (2004) might be confounded by processing differences for digits and letters. They produced evidence that digits yield higher spans than letters for both signers and speakers (Wilson & Emmorey, 2006a) and found no span differences for ASL and English for letters when they matched for articulatory duration and phonological similarity across modalities. Bavelier, Newport, Hall, Supalla, and Boutla (2006), however, called this matching process into question and produced further data in support of an intermodality span discrepancy, whose validity was in turn questioned by Wilson and Emmorey (2006b). Thus, evidence supporting an explanation of span differences between sign and speech in terms of articulatory duration or phonological similarity is inconclusive.

Retention of auditory and visual information.

The Bavelier group (Bavelier et al., 2006; Boutla et al., 2004) have pointed out the theoretical reliance of the phonological store on earlier sensory memory stores and the inherent modality-specific differences between these stores: auditory sensory memory traces persist for 2–4 s, whereas visual memory traces last at most 1 s. According to Boutla and colleagues, this may account for span differences.

Retention of temporal order information.

A further suggestion concerns potential differences in the retention of temporal order information across modalities. The auditory system is known to be adept at retaining sound order, whereas the visual system is less efficient in this respect, but better able to retain spatial structure. On the basis of this, it should be easier to retain the order of sound-based representations in working memory than their vision-based counterparts. This hypothesis is supported by evidence that shows that differences in performance on sign- and speech-based working memory tasks tend to disappear when the requirement for temporal order retention is removed (Boutla et al., 2004). Similarly, deaf children perform equally well on forward and backward span (Wilson et al., 1997), suggesting that item order is not a crucial aspect of temporary sign storage in working memory. Further, it has been found that the more deaf subjects rely on a speech-based code, the better they are at recalling temporal order (Hanson, 1982).

Neural Correlates of Working Memory for Sign Language

The neural correlates of working memory for sign language are largely similar to those of working memory for speech (Buchsbaum et al., 2005; Rönnberg et al., 2004; Rudner, Fransson, et al., 2007). However, there are also some neural regions engaged in working memory for sign language that are not involved to the same extent in working memory for speech (Rönnberg et al., 2004; Rudner, Fransson, et al., 2007). These regions include the occipitotemporal region bilaterally and the superior parietal region bilaterally. We suggest that the occipitotemporal involvement is related to identification, encoding, and storage of individual signs, whereas the superior parietal involvement may be related to the generation of a virtual spatial array for sign storage. These findings are in tune with behavioral evidence, which suggests largely modality-neutral storage and processing of working memory representations with some modality-specific aspects relating to sensory and perceptual aspects of processing information through the visuospatial medium of sign language (Rudner & Rönnberg, 2006).

The Working Memory Model for Ease of Language Understanding

The working memory model for Ease of Language Understanding (ELU, Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press) assumes sensory coding of language input but plays down the role of modality-specific aspects in working memory processing. Multimodal language input enters an episodic buffer that supports Rapid, Automatic Multi-modal Binding of PHOnology (RAMBPHO). As long as processing proceeds smoothly and there is no mismatch between input phonology and mnemonic representations, there is no need to engage modality-specific explicit processing mechanisms. Thus, the ELU model only predicts processing differences for sign and speech when processing is effortful. The role of explicit processing in the presence of mismatch was recently demonstrated in a study that showed increased explicit processing when the hearing aid settings of deaf participants were adjusted in a way that altered the phonological characteristics of the auditory input stream in relation to mnemonic representations (Rudner, Foo, Rönnberg, & Lunner, 2007, see also Rudner, Foo, Sundewall Thorén, Lunner, & Rönnberg, in press). Other evidence in support of the ELU model is provided by neuroimaging work that shows similar networks for perceptual processing of sign and speech with specialized components relating to sensorimotor artifacts of the modality in question (Emmorey et al., 2002, 2005; MacSweeney et al., 2002) and dissociated networks for sign language and nonlinguistic gesture (MacSweeney et al., 2004).

The ELU model predicts that modality-specific effects relating to working memory for sign language should only arise when processing is explicit, that is, when mismatch occurs due to phonological, capacity, or speed problems. Under these circumstances, language modality-specific differences may arise relating to the processing characteristics of the sensory modalities involved. Previous work has revealed that temporary storage in working memory for sign language appears to have a lower capacity and less temporal organization, under some circumstances, than equivalent storage for speech. On the basis of the ELU model, we propose that these circumstances are specifically associated with increased explicit processing demands.

This Study

In a set of three experiments, using three groups of participants DS, HS, and HN, we systematically investigated storage capacity and sensitivity to temporal organization in working memory for sign and speech under increasing explicit processing demands. We removed potential differences in retention of auditory and visual information by using one sensory mode of presentation only: nameable pictures. Dual Coding Theory (Paivio, 1991) provides a framework for understanding multimodal mental representation of nameable pictures and states that pictures are subject to representational processing, resulting in the direct activation of imagens (image generators). Referential processing then activates logogens (word generators) in the process of picture naming. In terms of Dual Coding Theory, in an experimental situation, easily nameable pictures directly activate imagens, in all sighted subjects, which in turn activate logogens in the appropriate language modality for signers and speakers. Thus, all subjects have similar imagens but logogens that are specific to their preferred language modality, sign, or speech. By using easily nameable pictures as stimuli, mental representations appropriate to the respective language modalities can be generated without undesirable sensory differences.

Although linguistic research has demonstrated that sign language has a phonological structure that parallels that of sign language, it has revealed differences in phonological patterning, including a relative emphasis on sequentiality for speech, that also seem to be reflected in neural organization (Emmorey, Mehta, & Grabowski, 2007). In this study, the inherent difference in sequentiality is demonstrated by the fact that whereas the speech labels of the picture stimuli have one to three syllables, the sign labels have only one or two movements. This unavoidable difference in phonological structure may, however, go some way towards counteracting potentially slower subarticulatory rehearsal for sign-based representations compared to speech-based representations (Bellugi & Fischer, 1972).

Phonological and semantic similarity are other factors that are known to interact with temporary storage capacity and sensitivity to temporal order, and these factors were also taken into account in the design of the experiments. Thus, having taken into account the retention of sensory information, articulation rate, and interitem similarity, we may assume that any intermodality differences in performance revealed in this study may be attributed to differences in retention of temporal order information.

The ELU model predicts that greater demands in terms of capacity and speed will lead to more explicit processing and thus a greater likelihood of modality-specific effects. In order to test this prediction, we tax capacity limits by presenting supraspan lists (8–9 items) and manipulate the speed of presentation and recall. ELU also predicts an effect of semantic similarity among to-be-remembered items as implicit episodic buffer processing in working memory involves access to semantic representations in long-term memory. In order to test this prediction, we manipulated semantic similarity among to-be-remembered items.

Experiment 1

Method

Participants.

Three groups (DS, HS, and HN) with nine persons in each took part in Experiment 1. There was no significant difference in age between the three groups (DS: X = 34, SD = 5.1; HS: X = 36, SD = 8.9; HN: X = 29, SD = 3.9). In all three groups, there were seven women and two men. All the DS were profoundly deaf and used sign language as their preferred language and all but one was prelingually deaf. Seven of the prelingually DS worked as sign language teachers and were thus proficient signers. The eighth prelingually DS came from a deaf family. The ninth DS became deaf before puberty as a result of neurological damage caused by infection and used sign language professionally on a daily basis.

All the HS reported normal hearing and all but three had started using sign language from birth as they had at least one parent or sibling who was deaf. These three had started using sign language in adulthood. All were regular users of sign language in their daily lives. All of them used sign language professionally as either sign language interpreters or teachers and thus were proficient signers.

All the hearing nonsigners reported normal hearing and either had no knowledge of sign language or only a rudimentary knowledge. All participants had normal, or corrected to normal, vision. All participants gave their informed consent.

Stimuli.

The stimuli were pictures of easily nameable objects selected from the worldwide web, which all had sign equivalents listed in the Swedish Sign Language (SSL) Dictionary (http://www.ling.su.se/). Inclusion in the SSL dictionary is currently the only objective measure of the frequency of SSL signs as no work has yet been done on examining the frequency of SSL signs. Pretesting involved three native Swedish speakers independently assigning lexical labels to the pictures. Where pictures could not be easily interpreted or where inappropriate labels were assigned, items were replaced and the procedure was repeated. This process resulted in a final set of pictures could easily be assigned appropriate lexical labels in Swedish. A native SSL user reviewed this list and confirmed that all pictures could easily be assigned appropriate lexical labels in SSL. The stimulus material was prepared in two separate versions with different, but equivalent material. In each version, the stimuli were arranged in eight lists of nine pictures each. No picture was repeated and no two pictures had the same lexical label. Thus, all in all, across the two versions there were 144 separate items.

We manipulated interitem similarity using four different list types: Distinct (no interitem similarity) and three list types with varying kinds of interitem similarity (Semantic, Speech, Sign). Each list type, apart from the Distinct lists, had two subtypes. The Semantic lists contained items that either belonged to the same semantic category (Subtype 1) or had the same physical shape (Subtype 2). The Swedish lists contained items whose Swedish lexical labels were phonologically similar by having the same place of articulation for either the first phoneme (Subtype 1) or the last phoneme (Subtype 2). The SSL lists contained items whose lexical labels in SSL were phonologically similar by having either the same handshape (Subtype 1) or the same handposition (Subtype 2). All the Swedish lexical labels applied to the pictures had between one and three syllables and the total number of syllables for all words in each list was between 14 and 16. Order of version was randomized, and order of list type and subtype within each version was randomized.

Presentation style.

A mixed temporal/spatial presentation style was used. This meant that list items were presented serially in a spatial pattern. The organization of temporary sign storage in working memory appears to be less dependent on temporal order information than equivalent speech-based functions (Wilson et al., 1997). Thus, a presentation style that places less emphasis on the temporal order of presentation may facilitate recall performance for deaf subjects. The nine items in each list were presented serially in a three-by-three matrix at 2-s intervals starting in the top left-hand cell and continuing from left to right and top to bottom. Once presented, each item remained visible until the ninth item had been visible for 2 s, when all items disappeared from the screen. This ensured that representation of spatial information could be built up during presentation but resulted in different presentation times for each list item.

Task.

The participants were instructed to memorize the items and their order and then recall the items by writing down the lexical label of the picture or drawing a representation of the item, in a response booklet. Each page of the response booklet contained a three-by-three matrix with nine cells, one for each list item, and the participants were instructed to recall items in the correct order and write or draw their responses in the correct matrix position.

Procedure.

All participants performed the task with all lists. The stimuli were presented on a PC using Powerpoint software. The participants were provided with a pen and a response booklet for recording their own responses. The participants were tested singly, at a computer screen, or in groups with the stimuli projected on a large screen. In all cases, participants were seated at a table with sufficient space for comfortable response recording. When tested in groups, the participants were instructed not to assist each other. Testing in groups facilitated data collection and did not compromise either stimulus presentation or recording of responses. Thus, it was considered equivalent to single testing.

When all the items in a list had been displayed, a question mark appeared in the center of the screen. This was the signal to the participants to recall the items. When the participants had finished recording their responses, they went on to the next list. This aspect of the procedure was self-paced.

Design.

The design was a 4 × 3 split-plot design. The within-group factor was list type (Distinct, Semantic, Speech, Sign). The between-group factor was sensory and linguistic experience (DS, HS, HN).

Data scoring and analysis.

All responses, written or drawn, that corresponded to pictures shown in the list concerned were scored as correct as long as they were recorded in the response booklet in the matrix cell corresponding to the cell in which it was originally presented. Thus, scoring reflects maintenance of order information as well as item information. Analyses of variance (ANOVAs) were computed for list type, list subtype, and group.

Results

The overall mean level of performance was 5.93 items per nine-item list (SEM = 0.18). Mean levels of performance for each group by list type and subtype are shown in Table 1. There was no significant difference in the level of performance between the three groups across list types. However, there was a difference in performance between list types across groups (F(3, 72) = 8.90, MSE = 2.07, p < .001, partial η² = 0.27). This main effect revealed a relative performance benefit for the Semantic lists, with a mean level of performance of 6.44 items per list compared to 5.75 items for Distinct lists (mean difference = 0.69, SEM = 0.19, p < .01, Bonferroni adjustment for multiple comparisons). There was no interaction between list type and group, suggesting that there was no difference in the levels of performance of the different groups on different list types.

Table 1.

Mean recall performance by list type and subtype for all three groups (HN, HS, DS) in Experiments 1 and 2

Group	List type	Subtype	Experiment 1 (n = 9)	Experiment 2 (n = 13)
			Mixed	Spatial	Serial
HN	Distinct	1	5.89 (0.40)	3.54 (0.44)	4.46 (0.44)
		2	5.50 (0.50)	4.00 (0.46)	3.92 (0.46)
	Semantic	1	6.83 (0.53)	4.31 (3.69)	4.39 (0.36)
		2	6.22 (0.28)	3.69 (0.51)	4.08 (0.44)
	Speech	1	6.89 (0.37)	3.69 (0.39)	3.92 (0.41)
		2	6.33 (0.47)	3.46 (0.47)	4.15 (0.29)
	Sign	1	5.61 (0.47)	3.46 (0.49)	4.08 (0.31)
		2	6.00 (0.45)	3.77 (0.47)	4.15 (0.47)
HS	Distinct	1	6.67 (0.40)	4.69 (0.44)	5.23 (0.44)
		2	5.56 (0.50)	4.08 (0.46)	4.00 (0.46)
	Semantic	1	7.00 (0.53)	5.46 (0.44)	5.15 (0.36)
		2	6.11 (0.28)	4.23 (0.51)	4.23 (0.44)
	Speech	1	6.00 (0.37)	4.92 (0.39)	4.46 (0.41)
		2	5.72 (0.47)	3.54 (0.47)	4.69 (0.29)
	Sign	1	6.06 (0.47)	4.23 (0.49)	4.23 (0.31)
		2	5.83 (0.45)	4.00 (0.47)	4.15 (0.47)
DS	Distinct	1	5.89 (0.40)	4.62 (0.44)	3.15 (0.44)
		2	5.00 (0.50)	3.46 (0.46)	3.54 (0.46)
	Semantic	1	6.78 (0.53)	4.77 (0.44)	5.08 (0.36)
		2	5.67 (0.28)	3.77 (0.51)	4.08 (0.44)
	Speech	1	5.89 (0.37)	3.69 (0.39)	4.31 (0.41)
		2	5.44 (0.47)	3.92 (0.47)	3.46 (0.29)
	Sign	1	4.94 (0.47)	3.39 (0.49)	3.69 (0.31)
		2	4.39 (0.45)	4.54 (0.47)	3.23 (0.47)

In this study, we manipulated the phonological similarity of the lexical labels of the picture stimuli within lists. Previous work has shown that phonological similarity among list items alters working memory performance for both sign (Wilson & Emmorey, 1997) and speech (Baddeley, 1966). However, the initial analysis did not reveal any effects of phonological similarity. Phonological sensitivity is a function of age of language acquisition (Mayberry & Eichen, 1991), and as some of the participants in the signing groups were not native signers, we performed a reanalysis of the data without the nonnative signers (one in the DS group and three in the HS group). Mean levels of performance for each group by list type and subtype are shown in Table 2. The reanalysis showed a similar main effect of list type as the initial analysis (F(3, 60) = 5.50, MSE = 2.09, p < .01, partial η² = 0.22), relating to a significant mean difference between performance on the Distinct and Semantic lists (mean difference = 0.63, SEM = 0.21, p < .05, Bonferroni adjustment for multiple comparisons) and in addition a significant interaction between list type and group (F(6, 60) = 2.73, MSE = 2.09, p < .05, partial η² = 0.21). Further investigation of this interaction revealed significant simple main effects of group for the Distinct lists (F(2, 20) = 21.63, MSE = 0.23, p < .01), Speech lists (F(2, 20) = 21.63, MSE = 0.23, p < .01) and Sign lists (F(2, 20) = 17.59, MSE = 0.23, p < .01) but not Semantic lists (see Figure 1).

Table 2.

Mean recall performance (SEM) by list type and subtype for all three groups (HN, HS, DS) in the reanalysis of Experiments (Exp) 1 and 2

Group	List type	Subtype	Experiment 1	Experiment 2
			Mixed	Spatial	Serial
HN: Exp 1, n = 9; Exp 2, n = 13	Distinct	1	5.89 (0.40)	3.54 (0.44)	4.46 (0.44)
		2	5.50 (0.50)	4.00 (0.46)	3.92 (0.46)
	Semantic	1	6.83 (0.53)	4.31 (3.69)	4.39 (0.36)
		2	6.22 (0.28)	3.69 (0.51)	4.08 (0.44)
	Speech	1	6.89 (0.37)	3.69 (0.39)	3.92 (0.41)
		2	6.33 (0.47)	3.46 (0.47)	4.15 (0.29)
	Sign	1	5.61 (0.47)	3.46 (0.49)	4.08 (0.31)
		2	6.00 (0.45)	3.77 (0.47)	4.15 (0.47)
HS: Exp 1, n = 6; Exp 2, n = 12	Distinct	1	6.58 (0.52)	4.66 (0.47)	5.33 (0.47)
		2	5.67 (0.66)	4.17 (0.50)	4.08 (0.48)
	Semantic	1	6.33 (0.66)	5.50 (0.43)	5.25 (0.38)
		2	6.17 (0.32)	4.25 (0.53)	4.33 (0.47)
	Speech	1	6.00 (0.48)	5.00 (0.40)	4.58 (0.42)
		2	5.33 (0.61)	3.67 (0.47)	4.75 (0.29)
	Sign	1	6.33 (0.61)	4.50 (0.48)	4.25 (0.30)
		2	6.00 (0.58)	4.00 (0.49)	4.33 (0.49)
DS: Exp 1, n = 8; Exp 2, n = 11	Distinct	1	5.75 (0.45)	4.73 (0.49)	3.18 (0.49)
		2	4.81 (0.57)	3.55 (0.52)	3.54 (0.51)
	Semantic	1	6.69 (0.57)	5.09 (0.45)	5.09 (0.40)
		2	5.75 (0.28)	4.18 (0.55)	4.09 (0.49)
	Speech	1	5.81 (0.41)	4.00 (0.42)	4.55 (0.44)
		2	5.38 (0.53)	4.00 (0.49)	3.64 (0.30)
	Sign	1	4.88 (0.53)	3.73 (0.51)	4.00 (0.31)
		2	4.19 (0.50)	4.91 (0.51)	3.46 (0.51)

Figure 1.

Interaction between list type and group in reanalysis of Experiment 1.

Discussion

There was no evidence of difference in overall performance between the three groups, DS, HS, and HN. However, there was an effect of semantic similarity, such that semantic similarity among list items had a facilitating effect on serial recall performance; this was true across groups. An effect of semantic similarity was predicted on the premises of the ELU model (Rönnberg, 2003; Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press). This model includes an episodic buffer that mediates matching of multimodal language input to representations in long-term memory. Thus, working memory representations have semantic features retrieved from long-term memory as well as semantic category information. The ELU model does not predict whether semantic similarity will enhance or interfere with recall and Baddeley's model plays down the effect of semantic similarity (Baddeley, 2003).

Although the results of the initial analysis in Experiment 1 did not reveal the effects attributable to language modality, the reanalysis, excluding nonnative signers, showed that native DS performed worse on Sign lists, which included items whose SSL labels either had a similar handshape or a similar position, than either of the two hearing groups. This finding is in tune with previous findings of poorer immediate serial recall for lists of phonologically similar signs by DS (Wilson & Emmorey, 1997).

The reanalysis also showed that HN performed better on the Speech lists, which included items whose Swedish labels had the same place of articulation for either the first phoneme or the last phoneme, than either of the two signing groups. Although phonological similarity among list items generally impairs immediate serial recall, it has been shown to enhance recall under some circumstances, for example, when recall of temporal order information is not required (Fallon, Groves, & Tehan, 1999; Fournet, Juphard, Monnier, & Roulin, 2003), when recall is delayed more than 8 s (Fournet et al., 2003) and for nonlexical items (Lian, Karlsen, & Eriksen, 2004). In this study, recall of order information was required, but because all items were visible simultaneously in a spatial array during encoding, it was theoretically possible to recall item order without recalling temporal order information. Total presentation time in this study was 18 s, and thus, item recall may have been delayed beyond the critical limit for a negative effect of phonological similarity. Easily nameable pictures may only be partially transformed into phonological code for rehearsal in working memory (Peters, Suchan, Zhang, & Daum, 2005), and thus, it is possible that items in this study in some sense were represented as nonlexical items but with sufficient phonological information (initial or final phoneme) to aid recall. All these three aspects of the design may contribute to positive effect of phonological similarity on memory performance for HN. Further, it is interesting to note that while the HS group performed like DS on the Speech lists, they performed like HN on the Sign lists. This suggests that HS are using speech encoding in preference to both sign encoding and visual encoding.

It is important to note that the discrepancy between the two analyses applies only to differences in sensitivity to phonological similarity between groups and not to semantic similarity. As the episodic buffer in the ELU model operates at the implicit level, no differences in processing between language modalities were expected. The difference in the pattern of results relating to phonological similarity between the main analysis and the reanalysis is probably due to the lower sensitivity to phonological patterning in sign language in the nonnative signers who were excluded from the reanalysis.

It is interesting to note that the phonological similarity effect was apparent despite lower n in the two signing groups. Although effect size was small (Cohen, 1977), it was in line with previous results and thus supports the validity of the test (Trusty, Thompson, & Petrocelli, 2004). This in turn supports the finding of no difference in performance between groups at the general level. Thus, it appears that the explicit processing demands in the experiment were not great enough to reveal intermodality differences in processing postulated by the ELU model.

In summary, the results of Experiment 1 showed an effect of semantic similarity for all groups as predicted by the ELU model in connection with implicit processing, and reanalysis, excluding late signers, revealed phonological similarity effects for both DS and HN. No difference in performance at a general level was revealed between groups.

In Experiment 2, we repeat Experiment 1 using new participants and with additional manipulations relating to explicit processing, temporal demands, and timing.

Experiment 2

Method

Participants.

There were three groups of participants with 13 persons in each group. The three groups were DS, HS, and HN, as in Experiment 1, but the participants were different. There was no significant difference between the ages of the three groups (DS: X = 35, SD = 12.4; HS: X = 35, SD = 10.5; HN: X = 37, SD = 14.4). DS were all native signers who were prelingually deaf, and students or staff at a folk high school where SSL is the teaching language and primary means of communication. All but two were profoundly deaf in both ears; one was profoundly deaf in one ear and had a moderate loss in the other ear, and the other had moderate to severe losses (60 dB) in both ears. All but two had started using SSL before the age of 3. These two had started using sign language at the ages of 7 and 8. The HS all reported normal hearing, and all but one had started using sign language from birth as they had at least one parent or sibling who was deaf. One HS had started using sign language at the age of 10.

The HS were all regular users of sign language in their daily lives. The hearing nonsigners all attended a folk high school. They reported normal hearing and either had no knowledge of sign language or only a rudimentary knowledge. The group of HN comprised eight women and four men, and the group of HS had the same gender composition. The group of DS comprised six women and six men. All participants had normal, or corrected to normal, vision. All participants had at least high school education.

Stimuli.

The stimuli were the same as in Experiment 1.

Presentation style.

Two different presentation styles were used, spatial and temporal presentation. With spatial presentation, the nine items in each list were presented simultaneously for a period of 9 s in a three-by-three matrix. With temporal presentation, the nine items in each list were presented serially for a period of 1 s each at the center of the computer screen, with no interstimulus interval. The two presentation styles were chosen to investigate the relative effect of temporal demands on performance by the three groups. The presentation time per item was reduced by one-half compared with Experiment 1 in order to increase time pressure and thus enhance conditions for explicit processing (Rönnberg, 2003). Item size was held constant over presentation styles.

Task.

The participants were instructed to memorize list items, and their order, and then recall the items by writing down the lexical label of the item or drawing a representation of it in a response booklet. There were two different types of response booklet, one for each style of presentation, and both contained eight pages, one for each list. For the spatial style, each page of the response booklet contained a three-by-three matrix with nine cells, one for each list item, and the participants were instructed to recall items in the correct order and write or draw their responses in the correct matrix position. For the temporal presentation style, each page had nine lines, one for each list item.

Procedure.

All participants performed the task with both styles of presentation and both versions of the material. Order of presentation style and material version was randomized. Order of list type and list subtype was randomized within each version. The lists of stimuli were presented using Superlab software on a portable PC. The participants were provided with a pen and a response booklet for recording their own responses. The participants were tested in pairs or singly at a computer screen. In either case, participants were seated at a table with sufficient space for comfortable response recording. When tested in pairs, the participants were instructed not to assist each other. Testing in pairs facilitated data collection and did not compromise either stimulus presentation or recording of responses. Thus, it was considered equivalent to single testing.

When all the list items had been displayed, a question mark appeared in the center of the screen. This was the signal to the participants to respond. When the participants had finished recording their responses, they went on to the next list. This aspect of the procedure was self-paced.

Design.

The design was a 2 × 4 × 3 split-plot design. The two within-groups factors were presentation style (spatial, temporal) and list type (distinct, semantic, speech, sign). The between-groups factor was sensory and linguistic experience (DS, HS, HN).

Data scoring and analysis.

Data scoring for the spatial style of presentation was performed as in Experiment 1. For the temporal style of presentation, all responses, written or drawn, that corresponded to pictures shown in the list concerned were scored as correct, as long as they formed part of a single sequence corresponding to all or part of the original presentation sequence. Analysis was performed as in Experiment 1. Due to a technical problem during data collection, data were lost for one DS and one HS for the condition Speech Subtype 1 with Temporal presentation. These missing data points were replaced with the mean score for the rest of the respective group for the particular condition. A serial position analysis of data obtained with the temporal presentation style was also performed. This analysis compared recall performance for presented items 1–3 (primacy), 4–6 (asymptote), and 7–8 (recency), irrespective of recall order.

Results

The overall mean level of performance in Experiment 2 was 4.11 items per nine-item list (SEM = 0.14). Mean recall performance by list type and subtype for each group and presentation style is shown in Table 1. There was no difference in performance between the two styles of presentation or between the three groups across list type and style of presentation. However, there was a main effect of list type (F(3, 108) = 5.17, MSE = 1.58, p < .01, partial η² = 0.13), as in Experiment 1, relating to superior performance on Semantic lists compared to Speech lists (mean difference = 0.42, SEM = 0.14, p < .05) and Sign lists (mean difference = 0.53, SEM = 0.13, p < .001; Bonferroni adjustment for multiple comparisons), although not Distinct lists. There was also an effect of list subtype (F(1, 36) = 17.61, MSE = 1.18, p < .001, partial η² = 0.33), whereby performance on Subtype 1 lists (semantic category, first phoneme, handshape) was superior to performance on Subtype 2 lists (physical shape, last phoneme, hand position). An interaction between list type and list subtype (F(3, 108) = 3.57, MSE = 1.78, p < .05, partial η² = 0.02) showed that the effect of list subtype was due to superior performance on list Subtype 1 (semantic category) with semantic lists (F(3, 108) = 23.62, MSE = 1.18, p < .01).

As in Experiment 1, some of the participants in the signing groups were not native signers, and thus, we performed a reanalysis of the data without the nonnative signers (two in the DS group and one in the HS group). The reanalysis showed a similar main effect of list type as the initial analysis (F(3, 99) = 4.37, MSE = 1.55, p < .01, partial η² = 0.12), relating to superior performance on Semantic lists. However, there was no interaction between list type and group.

The serial position analysis showed a main effect of serial position (F(2, 72) = 17.46, MSE = 1.68, p < .001, partial η² = 0.33) revealing a primacy effect (mean difference = 0.61, p < .001) and a recency effect (mean difference = 0.29, p < .05), in other words, a classic serial position curve. There was no difference between the serial position curves for the different list types or list subtypes or groups (see Figure 2).

Figure 2.

Serial position curve for recall with the serial style of presentation in Experiment 2.

To test whether the lack of interaction between list type and group in Experiments 1 and 2 was due to lack of power, we performed a metaanalyis of the data from Experiments 1 and 2.

Metaanalysis of Experiments 1 and 2.

Performance on Experiment 1 was significantly higher than on Experiment 2 (F(1, 60) = 68.73, MSE = 12.31, p < .001, partial η² = 0.53). There was no significant difference in performance between groups. There was a main effect of list type (F(3, 180) = 14.12, MSE = 1.77, p < .001, partial η² = 0.19) relating to superior performance on the Semantic lists compared to all the other list types (Distinct: mean difference = 0.53, p < .001; Speech: mean difference = 0.40, p < .01; Sign: mean difference = 0.74, p < .001; Bonferroni adjustment for multiple comparisons). There was also an interaction between list type and list subtype (F(3, 180) = 4.65, MSE = 1.86, p < .01, partial η² = 0.07) showing that the effect of list subtype was related to superior performance on list Subtype 1 (semantic category) with semantic lists (F(1, 65) = 29.26, MSE = 1.65, p < .01). However, there was no main effect of group and no interactions between group and any of the other variables. Reanalysis excluding nonnative signers repeated the pattern of significant difference in performance between studies (F(1, 53) = 49.63, MSE = 12.45, p < .001, partial η² = 0.48) and list types (F(3, 159) = 9.97, MSE = 1.75, p < .001, partial η² = 0.16). Moreover, there was a three-way interaction between Experiment, List type, and Group (F(6, 159) = 2.58, MSE = 1.75, p < .05, partial η² = 0.09).

Discussion

The results of Experiment 2 do not reveal any effects that can be attributed to language modality. There is no evidence of difference in overall performance between the three groups, DS, HS, and HN, and there is no evidence of any difference in performance between the three groups on individual list types. In Experiment 1, we found that exclusion of nonnative signers revealed group-specific effects relating to phonological similarity. In Experiment 2, there were no phonological similarity effects even though n was greater and even when nonnative signers were excluded. The metaanalysis confirmed that the phonological similarity effects revealed by Experiment 1 were not revealed in Experiment 2. Thus, the changes that were made between Experiments 1 and 2, increasing rate of presentation and manipulating the temporal aspect of presentation, did not result in any new modality-specific effects; indeed they counteracted the phonological similarity effects found in Experiment 1. The phonological similarity effect applies to working memory processing of both sign- and speech-based stimuli (Wilson & Emmorey, 1997), and thus, although it is tied to the processing of memory representations in different modalities, it is an effect that demonstrates the generality of working memory processing mechanisms, rather than modality specificity.

The changes made between Experiments 1 and 2 resulted in significantly poorer performance, with a moderate effect size (Cohen, 1977) indicating that the task was more difficult in Experiment 2 (irrespective of style of presentation) than in Experiment 1, and consequently, that more explicit processing was involved (Rönnberg, 2003). However, despite this, there was still no general difference in performance between groups. This applied even when the data from both experiments (66 subjects in all) were included in a metaanalysis. This suggests that increased speed and temporal demands in connection with presentation may not have been sufficient to reveal sign- or speech-specific effects postulated by the ELU model.

The finding of a semantic similarity effect in Experiment 1 was replicated in Experiment 2 and proved not to interact with presentation style. This suggests that the semantic similarity effect revealed in our data does not seem to be sensitive to timing and temporal aspects (cf. Rönnberg, Nilsson, & Ohlsson, 1982) and, thus, that this effect may not be as fragile as has been suggested in the literature (e.g., Baddeley, 1966). Haarmann and Usher (2001) reported data showing a facilitating effect of semantic similarity in immediate free recall, located at recency. However, in our data, the lack of interaction between serial position and list type suggested that the facilitating effect of semantic similarity was not specifically linked to recency. In addition, it was found that the semantic similarity effect is stronger when to-be-remembered items belong to the same semantic category than when they share shape characteristics. This suggests that the semantic similarity effect is based on a high level of inferential abstraction rather than a low level of visual perception. Thus, we can conclude that a facilitating effect of semantic similarity is a robust effect in immediate serial recall of easily nameable pictures. This effect is not affected by the preferred language modality of the participants or by the timing and temporal aspect of presentation. It is evenly distributed across list position and is related to high-level organizational principles. This suggests that the episodic buffer, which mediates semantic information stored in long-term memory, operates at an abstract level and in a similar manner irrespective of preferred language modality and experimental constraints. This supports the theoretical construct of RAMBPHO in the ELU model (Rönnberg, Rudner, Foo, & Lunner, in press).

The serial position analysis revealed a classic asymmetric bow-shaped serial position curve with a brief recency and more extended primacy effect (Murdock, 1974) that did not interact with group or list type. With auditory presentation, the recency effect usually approaches 100% (Murdock, 1974), but in our data, the recency effect is under 80%. This finding is in line with other work using nonauditory stimuli which show that the recency effect is weaker for visually presented stimuli (Conrad & Hull, 1968; Rönnberg & Ohlsson, 1980). This is known as the modality effect. Primacy and recency effects did not interact with group or list type, suggesting that the mnemonic processes involved are similar irrespective of preferred language modality or interitem similarity.

Although we stepped up the rate of stimulus presentation in Experiment 2, we did not alter the time constraints surrounding recall. In Experiment 3, we retain a stimulus presentation rate of 1 s/item and introduce a distracter test and a more controlled recall procedure. In this way, we increase explicit processing requirements, and thus, according to ELU model (Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press), improve conditions for revealing modality-specific effects. The recall procedure also allows for the separation of item and order recall, thus allowing us to analyze these two phenomena separately. The results of Experiments 1 and 2 showed interitem similarity effects relating to semantic category. In Experiment 3, we focus on the semantic similarity effect by using two list types only, Semantic (based on semantic category) and Distinct. We also focus on the two orthogonal groups, DS and HN and drop the HS group. However, we examine the effects of three different presentation styles, Temporal, Mixed, and Spatial, in one and the same experiment.

Experiment 3

Method

Participants.

There were two groups of participants with 14 persons in each group. The groups were DS and HN. In the DS group there were nine women and five men. In the HN group there were 4 women and 10 men. There was no significant difference in age between the groups (DS: X = 25, SD = 5.7; HN: X = 33, SD = 9.0).

All the DS were profoundly deaf and used sign language as their preferred language and all were prelingually deaf. All the hearing nonsigners reported normal hearing and either had no knowledge of sign language or only a rudimentary knowledge. All participants had normal, or corrected to normal, vision.

Stimuli.

The stimuli were easily nameable pictures (Snodgrass & Vanderwart, 1980). The stimulus material was prepared in three separate versions with different but equivalent material. Each version included eight lists of eight items each. No item was repeated. Thus, there were 64 separate items in each version. There were two different list types: “distinct” lists containing items that displayed minimal semantic interitem similarity and “semantic” lists containing items that belonged to the same semantic category. Half the lists in each version were distinct and half were semantic. An additional eight-item training list of distinct items was prepared.

Presentation style.

There were three presentation styles: spatial, temporal, and mixed. The three equivalent versions of the stimulus material were used for the three presentation styles. With all presentation styles, items were presented against a white background with two concentric circles forming a circular frame divided into eight cells (see Figure 3). In Experiment 3, a circular frame was chosen in preference to the square frame used in Experiments 1 and 2 so that all items were presented equidistant from the center of the frame. With the spatial style of presentation, items were presented simultaneously with one item per cell for 8 s. With the temporal style of presentation, items were presented one at a time for one second each, at center screen and with the mixed style of presentation, items were presented one at a time for 1 s each, starting in cell one and continuing to cell eight.

Figure 3.

Example of a Distinct list used in Experiment 3.

For each presentation style, each list type was presented four times with different material. Order of presentation style and list type was randomized.

Recognition.

The participants were instructed to memorize the identity of items presented, and their order, and to respond to recognition cues by striking one key for match and another for no match. Recognition was cued by presenting items one by one at center screen, surrounded by the presentation frame. For each list, five of the original list items were presented along with two novel items. Thus, recognition was probed for five out of eight or 62.5% of all items presented, and two out of seven or 28.6% of cues were lures. Each item was visible on the screen for 3 s or until the participant gave a correct response, indicating whether or not the item was one of the original list set. If the item was original, it remained visible along with a cross marking one of the cells of the presentation frame for another 3 s or until the participant gave a correct response, indicating item order. For the spatial and mixed styles of presentation, order match occurred when the cross appeared in the same cell in which the particular item was originally presented. For the temporal style of presentation, order match occurred when the cross appeared in the cell number equivalent to list position, for example, when the cross appeared in cell one for the first item on the list. Thus, order recall was probed for five items in each list. For half the lists, in each version recall was cued in cell number order and for the other half recall was cued in random order.

Distracter task.

In order to prevent rehearsal of the identity and order of presented items, a distracter task was introduced in Experiment 3. After each list had been presented for memorizing and before recall, a visuospatial distracter task was presented. Twenty rings were presented in four rows of five. Some of the rings (between three and seven) were formed by a solid line, whereas the others had a dotted line. The task was to determine whether the number of solid rings was even and in this case strike the match key, otherwise the no-match key. Three seconds was allowed for the distracter task.

Procedure.

All participants performed the task with all three presentation styles in balanced order, and list presentation order within each presentation style was randomized. The stimuli were presented on a PC using Superlab software. The participants were tested singly. A self-paced training session preceded each of the three presentation styles. Participants determined rate of progress between lists individually.

Design.

The design was a 2 × 2 × 2 split-plot design. The within-group factors were list type (distinct, semantic) and recognition cue order (serial, random). The between-group factor was sensory and linguistic experience (DS, HN).

Data scoring and analysis.

Responses and latency were registered automatically. ANOVAs were computed for accuracy and latency for item and order recognition for both groups.

Results

In contrast to Experiments 1 and 2, the design of Experiment 3 allowed separate analysis of performance relating to item identity and order.

Item recognition.

Out of the 10 items probed for the two replications of each eight-item list type, an average of 8.15 (SEM = 0.15) items were correctly recognized. Item recognition performance for all three presentation styles by recognition cue order and list type for the two groups is shown in Table 3. The level of performance did not vary significantly across group and there was no main effect on item recognition of the ordering of recognition cues. However, item recognition performance did vary with presentation style (F(2, 52) = 5.67, MSE = 2.55, p < .01, partial η² = 0.18). In particular, item recognition performance was lower with the spatial style of presentation compared to the two styles of presentation with a temporal aspect, the mixed (mean difference = 0.55, p < .05) and temporal (mean difference = 0.68, p < .05) styles of presentation (Bonferroni adjustment for multiple comparisons). Moreover, style of presentation interacted with order of recognition cues (F(2, 52) = 6.03, MSE = 1.45, p < .01, partial η² = 0.19) with serial cuing order having a facilitating effect that increases with the degree of temporal organization of the style of presentation (see Figure 4).

Table 3.

Mean recognition performance (SEM) for item identity and item order by list type and subtype for the two groups (HN, DS; n = 14) in Experiment 3

Recognition accuracy	Group	Recognition cue	List type	Mixed	Spatial	Serial
Item	HN	Serial	Distinct	8.14 (0.36)	7.14 (0.46)	8.64 (0.40)
			Semantic	8.71 (0.27)	7.71 (0.43)	9.14 (0.29)
		Random	Distinct	8.00 (0.33)	8.14 (0.38)	7.79 (0.45)
			Semantic	8.36 (0.39)	7.79 (0.39)	8.36 (0.39)
	DS	Serial	Distinct	7.64 (0.36)	7.43 (0.46)	8.14 (0.40)
			Semantic	9.07 (0.27)	7.50 (0.43)	8.71 (0.29)
		Random	Distinct	7.93 (0.33)	8.00 (0.38)	7.79 (0.45)
			Semantic	8.43 (0.39)	8.21 (0.39)	8.79 (0.39)
Order	HN	Serial	Distinct	4.29 (0.28)	3.21 (0.40)	3.86 (0.28)
			Semantic	4.21 (0.27)	3.71 (0.32)	3.93 (0.36)
		Random	Distinct	4.14 (0.33)	2.71 (0.31)	3.36 (0.37)
			Semantic	4.21 (0.33)	3.50 (0.35)	3.29 (0.34)
	DS	Serial	Distinct	3.79 (0.28)	3.07 (0.40)	3.86 (0.28)
			Semantic	4.07 (0.27)	3.79 (0.32)	3.07 (0.39)
		Random	Distinct	3.93 (0.33)	3.43 (0.31)	3.79 (0.37)
			Semantic	3.43 (0.33)	3.86 (0.35)	3.50 (0.34)

Note. Scores for item identity are out of 10 items cued across two eight-item lists. Scores for order are out of five items cued across two eight-item lists.

Figure 4.

Interaction between style of presentation and order of recognition cues in Experiment 3. Black blocks indicate serial presentation order of recognition cues and white blocks indicate random order.

Superior item recognition performance on the Semantic lists was revealed by a main effect of list type (F(1, 26) = 17.97, MSE = 1.17 p < .001, partial η² = 0.41). This effect did not interact with presentation style or group. Mean latency for item recognition was 1151 ms (SEM = 29.45). There was no main effect of latency between groups, but there was a tendency towards an interaction between list type and group (F(1, 26) = 3.95, MSE = 31,899.12, p = .06, partial η² = 0.13), which showed slower processing of Distinct lists than Semantic lists for DS (F(1, 26) = 5.24, MSE = 31899.12, p < .05) but not HN (see Figure 5).

Figure 5.

Interaction between list type and group in Experiment 3. Black blocks indicate Semantic lists and white blocks indicate Distinct lists.

Order recognition.

On average, correct order was recognized 3.67 (SEM = 0.14) times out of the five times it was cued for the two replications of each eight-item list type. Order recognition performance for all three presentation styles by recognition cue order and list type for the two groups is shown in Table 3. The level of performance did not vary significantly across group. However, there was a main effect of presentation style (F(2, 52) = 6.58, MSE = 1.62, p < .01, partial η² = 0.20). As was the case with item recognition, performance was poorest with spatial style of presentation, although here the difference was only significant in relation to the mixed style presentation (mean difference = 0.60, p < .01). There was no main effect of list type, but list type did interact with presentation style (F(2, 52) = 5.76, MSE = 1.01, p < .01, partial η² = 0.18), as a result of higher order recognition performance on Semantic lists with the spatial style of presentation (F(1, 26) = 6.93, MSE = 1.49, p < .05) (see Figure 6).

Figure 6.

Interaction between list type and presentation style in Experiment 3. Black blocks indicate Semantic lists and white blocks indicate Distinct lists.

There was no main effect of recognition cue order, but there was an interaction between cue order and group (F(1, 26) = 5.91, MSE = 0.52, p < .05, partial η² = 0.19), which showed that presentation order of recognition cues enhanced order recognition performance for HN (F(1, 26) = 9.04, MSE = 0.52, p < .05) (see Figure 7). Mean latency for order recognition was significantly slower for DS (1246 ms, SEM = 80.43) than for HN (1001.07 ms, SEM = 80.43) (F(1, 26) = 4.65, MSE = 1086740.33, p < .05, partial η² = 0.15).

Figure 7.

Interaction between recognition cue order and group in Experiment 3. Black blocks indicate serial order and white blocks indicate random order.

Discussion

The results of Experiment 3 revealed group effects. Although there was no significant difference in overall performance between the two groups DS and HN, there were significant interactions with the group variable relating to both item and order recognition. In particular, although order recognition accuracy was facilitated by serial order of recognition cues for HN, this was not the case for DS. Further, DS were slower to recognize items presented in Distinct lists than items presented in Semantic lists, whereas HN showed no difference in processing speed. DS were generally slower to recognize order than HN.

The ELU model (Rönnberg, 2003; Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press) predicts effects of language modality when processing becomes explicit as a result of mismatch induced by problems with phonology, capacity, or speed. In Experiment 3, explicit processing demands were increased compared to Experiments 1 and 2. A distracter task was introduced between encoding and retrieval, and time constraints were introduced in connection with memory retrieval. The response mode was also changed from pen and paper in Experiments 1 and 2, which may place different demands on groups with different signing/speaking preferences, to yes/no key presses. Thus, conditions were optimized to reveal working memory processing and storage differences between habitual signers and habitual speakers. Under these optimized conditions, differences were revealed between the two groups.

Both hearing and deaf groups demonstrated a sensitivity to order of presentation of “item” recognition cues, but only the hearing group demonstrated a sensitivity to order of presentation of “order” recognition cues. Sensitivity to order of recognition cues relative to presentation order indicates that order of encoding is stored in working memory. Thus, results suggest that order of encoding is stored in working memory for both groups. This was apparent from the even performance across groups on immediate serial recall in Experiments 1 and 2. In particular, the similarity of serial position curves across groups in Experiment 2 indicated concurrent encoding processes across groups. However, the difference in sensitivity to order recognition cues in Experiment 3 indicates differences in the way DS and HN process order information in working memory.

For both groups, item and order performance was poorest when the presentation style had no temporal aspect. Further, serial order of item recognition cues facilitated performance of both groups when the presentation style had a temporal aspect, whereas serial order of order recognition cues only facilitated performance of the hearing group, and did so irrespective of presentation style. Previous work has shown that temporary storage in working memory for sign language compared to speech seems to be less dependent on temporal order information (Wilson et al., 1997), and thus, we predicted that a presentation style that places less emphasis on the temporal order of presentation may facilitate recall performance for deaf participants. The results of this study show that temporal order of presentation facilitates encoding and retention of item information in a similar way for DS and HN. When it comes to recognition of order, serial cuing order gives HN an advantage. We have suggested previously that working memory representations of signs may be stored in a virtual spatial array in working memory (Rönnberg et al., 2004). Such an array may allow equal ease of access to any position in the array, in contrast to an auditory-based loop in which representations can only be accessed in serial order. The finding of an advantage of serial ordering of order recognition cues for HN, but not DS, in this study is in line with this suggestion.

The results of Experiment 3 also showed that DS are generally slower at order recognition than HN. This finding further supports the idea of a random access virtual spatial array for sign language. In Experiment 3, order recognition cues were presented in serial order in half of the trials. This means that order recognition was facilitated and, thus, presumably speeded up for HN in half of the trials. However, serial presentation of cues never facilitates order recognition for DS, and thus it is never speeded up.

Between-group effects were small (Cohen, 1977). However, their reliability is supported by the fact that they could be explained in terms of our ELU-based predictions and previous empirical work.

The results of Experiments 1 and 2 showed that semantic similarity among to-be-remembered easily nameable pictures enhances immediate serial recall performance, irrespective of hearing and sign language ability. The results of Experiment 3 showed that the semantic similarity effect is related to retention of both item and order information. Semantic similarity among list items enhanced accuracy of item recognition for both groups, and for DS, it enhanced speed of item recognition. As regards order recognition, semantic similarity enhanced performance with the spatial style of presentation across groups. The ELU model (Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press) postulates an episodic buffer, which mediates binding of input to long-term memory representations. The results of this study show that working memory representations based on easily nameable pictures include semantic category information and that this information can be used as a cue in connection with memory retrieval. These general findings apply irrespective of hearing and sign language ability. The results of Experiment 3 show, more specifically, that it is in particular item retrieval that is enhanced and that this is particularly the case for DS, whose recognition speed is improved. More surprisingly, we find that semantic similarity can also facilitate order recognition if other organizational principles (e.g., temporal aspects) are lacking. This suggests that semantic similarity provides a basis for organization of the content of working memory.

Conclusion

This study addressed the issue of whether increasing explicit demands on working memory processing for sign language result in lower capacity and less temporal organization than comparable working memory processing for speech-based language. Results of all three individual experiments with 27, 39, and 42 participants, respectively, and a metaanalysis of Experiments 1 and 2 showed that working memory for easily nameable pictures is highly similar for DS, HS, and HN. This supports the notion that previously identified differences in the capacity of temporary storage in working memory for sign and speech may be due to differences in retention of auditory and visual information (Boutla et al., 2004), which were held constant in this study.

Between-group performance differences emerged in Experiments 1 and 3. In Experiment 1, when explicit processing demands were low, differences related to a phonological similarity effect for both DS and HN, emphasizing similarities in working memory processing across modalities. In Experiment 3, when explicit processing demands were high, differences related to the organizational principles of temporary storage in working memory. Working memory storage seems to support temporal information in both hearing and deaf participants. Working memory retrieval, however, does not seem to have a temporal bias for the deaf group. This finding supports and extends previous findings (Rönnberg et al., 2004; Wilson et al., 1997). The fact that these group differences emerged when explicit processing demands were high is in line with the ELU model (Rönnberg, 2003; Rönnberg, Rudner, Foo, & Lunner, in press; Rönnberg, Rudner, & Foo, in press). The pattern of semantic similarity effects extends the ELU model by suggesting that the episodic buffer, which mediates semantic information stored in long-term memory, operates at an abstract level and in a similar manner irrespective of preferred language modality and experimental constraints.