Experimental Semiotics: A Review

Abstract

In the last few years a new line of research has appeared in the literature. This line of research, which may be referred to as experimental semiotics (ES; Galantucci, 2009; Galantucci and Garrod, 2010), focuses on the experimental investigation of novel forms of human communication. In this review we will (a) situate ES in its conceptual context, (b) illustrate the main varieties of studies thus far conducted by experimental semioticians, (c) illustrate three main themes of investigation which have emerged within this line of research, and (d) consider implications of this work for cognitive neuroscience.

Conceptual Context of ES

Given its focus, experimental semiotics (ES) is in close relation with three well-established lines of research. The first of these is the experimental investigation of spoken dialog (e.g., Krauss and Weinheimer, 1966; Clark and Wilkes-Gibbs, 1986; Garrod and Anderson, 1987; Horton and Keysar, 1996) which for convenience we refer to as experimental pragmatics¹ (Noveck and Sperber, 2006). Experimental pragmatics has produced a wealth of knowledge about language use (e.g., Clark, 1996) and continues today to provide important insights into human communication (e.g., Brennan and Hanna, 2009; Garrod and Pickering, 2009; Shintel and Keysar, 2009). ES is similar to experimental pragmatics in at least two regards. First, both approaches aim at uncovering the causal relations behind the phenomena they observe by using methods that afford experimental manipulation and control. Secondly, they both share the assumption that, in order to understand human communication, researchers must investigate human social interactions as well as individual cognitive processes. Despite these similarities, however, ES differs in a fundamental way from experimental pragmatics because it focuses on interactions that occur in the absence of pre-established communicative conventions². In other words, experimental semioticians study novel forms of communication which people develop when they cannot use pre-established communication systems. As we shall see in Section “Main Themes of Investigation in ES,” this is a difference which makes a difference.

The focus of experimental semioticians on novel forms of communication leads us to the other two lines of research which are in close relation with ES: natural experiments with human languages and computer simulations. As for the former, the emergence of novel forms of human communication has been extensively investigated through sign languages that emerged in relatively isolated populations (e.g., Kegl, 1994; Sandler et al., 2005) as well as through home sign systems spontaneously developed by deaf children raised by non-signing parents (e.g., Goldin-Meadow and Feldman, 1977; Goldin-Meadow and Mylander, 1998). These lines of research have produced a wealth of knowledge about the origins of novel languages (Kegl et al., 1999; Goldin-Meadow, 2003). However, because experimental semioticians observe the emergence of communication in the laboratory, they gain access to new opportunities for scientific inquiry. One such opportunity is having access to the complete history of the development of a communication system. The details of this history can help us understand the processes that lead humans to communicate successfully. For example, four studies in ES (Galantucci, 2005; Garrod et al., 2007; Healey et al., 2007; De Ruiter et al., 2010) reported that signs established later on during the development of a novel communication system often incorporated parts of previously established signs. This occurred with communication systems which comprised only a handful of signs, suggesting the hypothesis that linguistic structures may emerge in the very early stages of a communication system. As we shall see in Section “Early Emergence of Linguistic Structure,” further studies by experimental semioticians tested this hypothesis at different linguistic levels and found supporting evidence for it (Selten and Warglien, 2007; Kirby et al., 2008; Galantucci et al., 2010; Theisen et al., 2010). Without the complete record of the development of a communication system, the hypothesis would have been difficult to test. In addition, experimental semioticians can carry out manipulations that would be difficult to realize outside the laboratory. For example, Fay et al. (2010) systematically manipulated the composition of the communities of people in their study, Galantucci et al. (2010) manipulated the rapidity of fading of the medium people used to communicate, and Selten and Warglien (2007) manipulated the number of symbols people could use in their game. Outside the laboratory, such manipulations would pose insurmountable ethical and practical challenges. Of course the move from the natural context to the laboratory is not without costs as it entails a loss in ecological validity with respect to natural experiments. This loss and its implication for the scientific relevance of ES are discussed in Section “Conclusion.” Here we argue that, in spite of the loss, ES might still aspire to capture essential properties of the communication systems which develop in natural contexts. We see two obvious differences between natural experiments and current ES studies. First, the two operate over vastly different timescales. Second, ES studies focus on communication systems developed by adults, while natural experiments focus on communication systems developed primarily by children. At the present we do not know the extent to which these differences limit ES. Should the limitations turn out to be substantial, experimental semioticians can address them by increasing the timescale of their studies or, as we suggest in Section “Conclusion,” by realizing them with children.

Because of the opportunities it offers for experimental manipulation, ES also relates to research on computer simulation of the emergence of communication among artificial agents (Steels, 1997). This line of research has offered important insights into how novel communication systems might emerge (e.g., Hurford, 1989; Quinn, 2001; Cangelosi and Parisi, 2002; Kirby, 2002; Skyrms, 2002; Steels, 2003) and how they might evolve over time (e.g., Clark and Roberts, 1993; Hare and Elman, 1995; Briscoe, 2000; Smith et al., 2003). However, although current simulations are designed to model ever richer aspects of human behavior (e.g., Steels and Wellens, 2006; Vogt and Divina, 2007) there remains a wide gulf in behavioral complexity between artificial agents and humans. In other words, drawing inferences from simulations to natural human phenomena is often problematic. ES provides an ideal source of complementary knowledge to that provided by simulations.

In the next section we introduce ES in more detail, illustrating the main varieties of studies that contributed to its recent growth.

Main Varieties of Studies in ES

Despite its brief history, ES has already developed in a number of different directions. In this section we survey the studies that contributed to this development with the main goal of identifying the different varieties of experimental paradigms that have emerged. In particular, we will focus on the methodological differences among these paradigms; some of the specific results of the studies surveyed here are illustrated in Section “Main Themes of Investigation in ES.”

Semiotic referential games

The earliest studies in ES were carried out by Healey et al. (2000, 2002, 2004), who used a graphical medium to study the development of novel communicative conventions among pairs of individuals³. These researchers adopted standard referential communication tasks such as those used in experimental pragmatics (e.g., Krauss and Weinheimer, 1966) but prevented the use of spoken or written language. In particular, they had participants repeatedly draw a stimulus such as a piece of music or a concept for a partner to identify, without using letters or numbers (we will refer to these tasks as semiotic referential games). Over a number of trials, Healey and colleagues observed partners developing spontaneous communicative conventions to succeed at the task. The development of such conventions has been extensively investigated in the last few years, through manipulations such as the type of interaction in the game (Garrod et al., 2007; Healey et al., 2007) or the social organization of the community of players that participated in the game (Garrod et al., 2010). An interesting result from these studies was that the increased simplification and evolution of graphical forms depended crucially on communicators’ ability to give graphical feedback to their partner. For example, when other participants repeatedly drew the same concepts for an imaginary audience, the drawings became increasingly complex with repetition. Furthermore, overseers (i.e., people who saw the drawings produced by an interacting pair but could not interact with them) were much poorer at understanding the graphical signs than those directly involved in the interaction. These findings are reminiscent of those from referential communication tasks using natural language (e.g., describing “tangram” figures), in which simplification of descriptions depends on feedback (Krauss and Weinheimer, 1966) and “overhearers” are much poorer at identifying the referents of descriptions than are the original participants (Schober and Clark, 1989).

Two other findings from these studies, which we discuss in more detail in Sections “Symbols Arising from Social Interaction” and “Diffusion of Novel Communication Systems Within and Across Generations of Users,” are (1) that with repeated interaction the forms (i.e., drawings) systematically change to become more abstract and symbolic, and (2) that networks of players develop their own conventionalized sets of forms reflecting the common history of interaction within the community. Despite slight differences in methods and research focus, studies performed with semiotic referential games all share the important feature that, whereas the set of forms that people use for communication is open (i.e., they can draw what they like) the set of referents to communicate in the game (e.g., concepts or pieces of music) is typically closed and pre-determined by the experimenter. This choice is well suited for studying the emergence and evolution of signs but, as we shall see in a moment, it is not the only possible methodological choice in ES.

Coordination semiotic games

A number of researchers (Galantucci, 2005; Noordzij et al., 2009; Scott-Phillips et al., 2009) have introduced tasks that involve more severe semiotic challenges than referential semiotic games by asking people to develop shared referents as well as shared communicative forms (we will refer to these tasks as coordination semiotic games). In other words, whereas referential semiotic games typically assign a set of pre-established referents to be communicated, coordination semiotic games leave players free to discover referents that support success in the game. Players of a coordination semiotic game move an agent in a virtual space with the overt goal of coordinating the moves with a partner. Crucially, achieving the goal depends on successful communication. However, successful communication can be supported by different sets of referents (cf. Garrod and Anderson, 1987) and, in consequence, coordination semiotic games require players to converge on a common choice of referents as well on a common set of forms to indicate the referents. For example, in the game developed by Galantucci and colleagues (Galantucci et al., 2003; Galantucci, 2005) coordination could be achieved either by referring to concrete spatial markers in the game environment or by referring to abstract geographic coordinates. In such conditions, converging on a shared set of referents is as important as developing forms to identify specific referents within the set. In fact, the semiotic challenge in coordination semiotic games is so severe that sometime participants perform very poorly or fail at the task (Galantucci, 2005; Scott-Phillips et al., 2009). As we shall see in Section “Insights from Failures,” such failures provide useful information about the necessary ingredients for the emergence of communication (Galantucci and Steels, 2008; Galantucci, 2009).

Additionally, coordination semiotic games typically require players to communicate through fairly unusual means. For example, participants in the game developed by Galantucci and colleagues communicated through a graphical medium in which visual signals had a short permanence (similar to speech) and reflected only the horizontal component of the participants’ drawings (see Figure 1A). In such conditions, the possibility of using pictorial representations or well-established graphical symbols is greatly reduced (Figure 1B).

Figure 1.

(A) How the drawings players produced on the digitizing pad appeared on the screen. (B) How common graphic symbols drawn on the digitizing pad appeared on the screen. (Adapted from Galantucci et al., 2010).

Other researchers (de Ruiter et al., 2007; Scott-Phillips et al., 2009) eliminated altogether the presence of a medium specifically dedicated to communication. In the coordination games developed by these researchers, players had to craft communication forms using the very actions that constituted moves in the games. In other words, participants in these studies had to find ways to signal that some of the moves that their agents performed had the intent of communicating rather than the intent of bringing the agent to a specific point in the game environment.

Semiotic matching games

The games discussed so far are useful for studying the emergence and early evolution of communication systems but, because players often succeed at these games with relatively simple systems, they are not ideal for studying how sophisticated forms of language-like structures might emerge and evolve. For this purpose, a number of researches (Selten and Warglien, 2007; Kirby et al., 2008; Roberts, 2008) softened the challenge typical of semiotic referential games, providing players with a closed set of communication forms as well as a closed set of referents (we will refer to these tasks as semiotic matching games). In particular, Selten and Warglien (2007) provided pairs of participants with a set of letters which they had to combine to communicate about a closed set of geometric figures while Kirby et al. (2008) and Roberts (2008, 2010) asked people to memorize a pre-established mapping between artificial words and a closed set of referents. While they differ in terms of the linguistic forms used and the social processes studied (Selten and Warglien focused on dyads, Kirby and colleagues on chains of individuals, and Roberts on competing groups of players), these studies all share an important feature. Thanks to the use of closed sets of communication forms, it is possible to create simple measures of language-like structures. In particular, the former two studies used these measures for detecting the emergence of compositionality (see “Emergence of Compositionality”), while the latter used them to study the emergence of subtle linguistic variations that distinguished different social groups (see “Role of social dynamics during the development of a communication system”).

Main Themes of Investigation in ES

In the next three sections we illustrate the main themes of investigation which have so far emerged within ES. Here we briefly introduce each of the sections, highlighting the scientific relevance of the themes they present.

The first section focuses on the early emergence of linguistic structure. This theme relates to one of the core ambitions of ES, which is to discover the extent to which key features of natural languages (Hockett, 1960) arise from general principles of human communication rather than specific characteristics of those languages. As we shall see in Section “Early Emergence of Linguistic Structure,” Experimental Semioticians have begun to provide evidence that this is the case. Such evidence is difficult to obtain from field studies with spontaneously emerging natural languages (e.g., home sign or Nicaraguan Sign Language) because they do not permit experimental manipulations; neither can it come from experimental pragmatics because of its reliance on the use of already established languages.

The second section focuses on the effects that different types of social manipulations have on emerging communication systems. A number of such manipulations have been carried out before in experimental pragmatics (e.g., Schober and Clark, 1989; Garrod and Doherty, 1994; Fay et al., 2000). However, because these studies involved participants conversing in their own language, it is not clear to what extent the findings reflect specific features of conversation or general principles of social interaction and communication. Again, ES studies test the generality of these findings.

The third section focuses on the emergence of communication, that is, on the processes that enable people to create a communication system from its very foundations. The possibility to study such processes in the laboratory is a unique opportunity offered by ES which circumvents two inherent limitations of field studies with spontaneously emerging natural languages. The first limitation is that natural languages become available for scientific inquiry only after they have emerged. At that point, the early stages of the emerging languages are very difficult to reconstruct. The second limitation is that novel natural languages emerge rarely. In contrast, experimental semioticians can observe the emergence of novel forms of human communication as often as needed and have full access to the history of such emergence.

Early emergence of linguistic structure

As remarked in Section “Conceptual Context of ES,” the communication systems people develop in ES studies tend to quickly adopt key design features of human languages. In particular, researchers have shown that these systems exhibit combinatoriality and compositionality. These two features are particularly important for human communication because, together, they give rise to one of the key hallmarks of human language, duality of patterning (Hockett, 1960; Martinet, 1960).

Emergence of combinatoriality

All of the meaningful units of human language can be generated by combining a few dozen meaningless forms. Thanks to this property – here referred to as combinatoriality – languages can express widely different meanings by combining the same forms in slightly different ways. For example, the English words /ækt/ (act), /kæt/ (cat) and /tæk/ (tack) are all composed by recombining the same forms, the phonemes /k/, /æ/ and /t/. Is there evidence for the emergence of combinatoriality during the evolution of non-linguistic communication systems?

Using a semiotic coordination game, Galantucci et al. (2010) found that novel communication systems can exhibit a considerable degree of combinatoriality, from their early inception. They analyzed 14 communication systems developed by different pairs (comprising on average about nine signs) and measured the combinatoriality of these systems by counting the number of times basic forms recurred across the signs of each system (for the details of the method, see Galantucci et al., 2010). The result was clear: Across the 14 systems, basic forms recurred very frequently, suggesting that novel communication systems quickly adopt a combinatorial design.

Moreover, the degree of combinatoriality of the communication systems was affected by the rate of fading of their forms. Communication systems developed over a rapidly fading medium were more combinatorial than systems developed over a medium that faded more slowly (Figure 2). In other words, by manipulating rate of fading, Galantucci and his colleagues uncovered one of its hidden effects, providing an example of how ES can lead to novel insights into the design of human communication systems.

Figure 2.

Examples of the sign systems studied by Galantucci et al. (2010). The signs are placed in the location on the game map they indicated. (A) Non-combinatorial system developed for the slowly fading medium. Each sign is composed of a distinct basic form. (B) Highly combinatorial system developed for the rapidly fading medium. Every sign contains at least one basic form that is present in one other sign. Forms are sequenced in time. (Adapted from Galantucci et al., 2010).

Considering that human natural languages typically rely on rapidly fading forms (e.g., patterns of sound or dynamic gestures), the study by Galantucci et al. (2010) suggests a new explanation for their high degree of combinatoriality.

Emergence of compositionality

One of the defining characteristics of the semantics of natural language is compositionality, by which complex expressions take their meaning from the combined meanings of their parts. For example, the meaning of the complex expression “red house” is derived from combining the meanings of its components “red” and “house,” hence a “red house” is a house that is red. Is there evidence for the emergence of compositionality during the evolution of non-linguistic communication systems?

One of the first experiments showing such emergence was by Selten and Warglein (2007). Pairs of participants played a semiotic matching game in which they had to match strings of letters to visual tokens (different shapes of different colors containing different inserts; Figure 3 top row) in such a way that both players only succeeded if they made the same assignments. Over the course of many trials players came to develop common code systems, with the same mappings between the strings and the visual tokens. The question is whether these code systems exhibited compositionality; such a code might look like that in Figure 3 (bottom row), in which the letter sequences R, RM, and RZ are matched with figures such that R stands for circle, M for o insert and Z for + insert. Interestingly, although some players did develop compositional code systems and were especially successful with them, such systems only emerged when there was a wide range of different letters to use with a large number of tokens (e.g., when players were faced with situations in which they often had to encode novel meanings).

Figure 3.

Examples of figures and code systems from Selten and Warglien (2007). (Top row) Example of six visual tokens made up of two shapes and three inserts. (Bottom row) Example of a compositional grammar in which R signifies circle, S signifies triangle, M signifies • and Z signifies +. (Adapted from Selten and Warglein, 2007).

Other studies also suggest that the emergence of compositional code systems depends on the task context. Kirby et al. (2008) used a different iterated learning paradigm to investigate how non-linguistic code systems might evolve when passed down generations of learners (for an overview of this line of research, see Cornish, 2010). In their task Player 1 learns an “alien” language made up of random pairings of written symbols (sequences of letters) with visual stimuli (colored objects in motion), he is then tested on both previously encountered and new symbols (i.e., combinations not previously seen) and the results of this test then become the learning set for Player 2. Player 2 then learns Player 1’s version of the “alien” language, which is then passed on to Player 3 in the same fashion. This process iterates through a chain of 10 players. As with other iterated learning tasks, the languages changed as they were passed down the chain. But the nature of the resulting languages depended on what could be transmitted from player to player. If there was no interference in what was transmitted, chains settled on very ambiguous languages without compositionality. For example, whereas the first player was given 14 distinct word-meaning pairings to learn, the last player would typically use only four words to cover all 27 meanings. However, if at each transition all but one ambiguous words were removed from the learning set, then more expressive languages emerged with around 16 words in the last player's vocabulary. Furthermore, these more expressive languages commonly exhibited compositionality (see Figure 4).

Figure 4.

Example of a partially compositional grammar elicited by Kirby et al. ’s (2008) iterated learning task. The string associated with a picture consists of substrings expressing color (n, l, r), shape (ere, eho, eki), and motion (ki, plo, pilu) respectively. The hyphens were added for clarification and were not present in the words produced by the original participants. (From Kirby et al., 2008).

The emergence of compositionality has also been demonstrated using a semiotic referential game. Theisen et al. (2010) used a version of Garrod et al. (2007) “pictionary” task to investigate compositionality. In this task players have to communicate different concepts chosen randomly from a fixed list by drawing them (see “Semiotic Referential Games”). Although, earlier experiments using this task had shown some evidence for the emergence of compositionality in the drawings (Garrod et al., 2007, pp. 984–985), it was not very prevalent. Therefore, Theisen and colleagues decided to manipulate the semantic structure of the sets of concepts to be communicated. For example, they included different agricultural concepts (farmer, barn, farming, tractor) and a cross-cutting set of educational concepts (teacher, school, teaching, school bus). And, it turned out that their participants developed graphical signs exhibiting compositionality. For example, all signs relating to school concepts might include a “chair” emblem (see Figure 5).

Figure 5.

Example of drawings elicited by Theisen et al. ’s (2010) pictionary experiment that illustrate partial compositionality. All educational concepts include a small chair token, all academic concepts include a mortar board like token and so on. (Adapted from Theisen et al., 2010).

So compositionality of the kind exhibited in natural languages can emerge in sign systems evolved in the laboratory. However, to date semiotic experiments have only demonstrated the emergence of certain kinds of compositional structure. For example, Selten and Warglein (2007) never observed “positional” compositionality, in which the same sign takes on systematically different interpretations depending upon its position in the sequence (e.g., in the Arabic number system 1 in 01 has a different interpretation from the 1 in 10). This was why they never observed compositional codes if the participants in their experiments only had access to two letters for their set of tokens⁴. As far as we are aware, such positional compositionality never emerged from Kirby et al.’s (2008) or Cornish et al.’s (2010) experiments either. So this is a challenge for future work.

Social manipulations in ES

One of the core assumptions of ES is that, in order to understand human communication, researchers must investigate human social interactions as well as individual cognitive processes. Indeed, as we remarked above, experimental semioticians have found that differences in the type of social interactions allowed in their studies can have profound effects on the development of novel communication systems. Experimental semioticians have investigated social interactions from three different perspectives. The first one focuses on different types of dyadic interactions. The second one focuses on the social diffusion of novel communication systems, both within a generation of users and across generations of users. The third perspective focuses on the effects that in-group and out-group dynamics can have on the development of novel linguistic conventions.

Symbols arising from social interaction

One of the great challenges in understanding the evolution of human communication systems is the emergence of symbols, in which the relationship between the sign and what it stands for is purely arbitrary. For example, how might symbolic words, such as “dog” or “chien” (for canine), have emerged? Unlike iconic signs, the sound of the words bears no relation to the sounds of the creature they stand for, nor is there any relationship between the sounds of the words themselves⁵. Does ES give us any clues to understanding the emergence of symbols?

Semiotic referential games offer interesting clues. The first of these comes from Garrod et al.’s (2007) “Pictionary” task (see “Semiotic Referential Games”). They had participants communicate a series of easily confusable items (e.g., Microwave, television, computer monitor) by drawing on a standard whiteboard. In one condition pairs communicated the recurring items altern between drawing and matching roles from one game to the next. The changing form of the signs used to convey “computer monitor” across six games of the task is shown in Figure 6. What started out as an iconic depiction of the computer monitor and keyboard rapidly evolved into a simplified symbolic form (inverted U shape). We can tell that the more abstract form is symbolic because viewers, when shown the sign out of context, do not recognize that it stands for computer monitor. Crucially, such refinement only occurred when partners were allowed to interact graphically (even if this only involved placing a tick next to the drawing to indicate comprehension). In a control condition in which the participants simply repeated their drawings for an imaginary audience the drawings became more complex with repetition and retained their iconic character. A second finding was that with extended interaction communicators’ drawings became increasingly similar, or convergent (See blocks 5 and 6, Figure 6).

Figure 6.

Drawing refinement and convergence for the concept “Computer monitor” across six games between a pair of interlocutors playing the interactive version of the Pictionary task (adapted from Garrod et al., 2007).

Thus, in this task symbolic graphical communication systems evolved through a process of interactive grounding. Specifically, Garrod et al. (2007) argued that icons rapidly evolve into symbols via interaction; icons help ground shared sign systems and interaction promotes a shift in the locus of information from the sign to the users’ memory of the sign's usage. This shift in information facilitates the evolution of increasingly simple abstract signs that are easy for communicators to produce and interpret.

We should note that this interactive evolution from complex iconic graphical signs to simpler symbolic graphical signs parallels findings from experimental pragmatic studies of referential communication. In a tangram description task Brennan and Clark (1996) found that interlocutors often developed idiosyncratic short-hand descriptions for the tangram figures (e.g., ice-skater to refer to one of the shapes). They argued that these idiosyncratic descriptions reflected conceptual pacts established between the communicators through an interactive grounding process similar to that assumed above. Of course, although conceptual pacts lead to descriptions that are simpler than their precursors, as happens in the evolution of graphical signs, such descriptions are no more symbolic than their precursors. There is still some controversy as to the exact mechanisms responsible for this process in linguistic communication (see Garrod and Pickering, 2007) but the parallel between the graphical and linguistic changes suggests that the process reflects a general property of interactive referential communication rather than a purely linguistic property.

The results from other referential semiotic studies also highlight the importance of the nature of the graphical interaction in establishing different kinds of signs. Healey et al. (2007) manipulated the nature of the feedback that communicators could use during their music-matching task (see “Semiotic Referential Games”). In one condition (concurrent drawing) participants were able to simultaneously draw on the shared white board as they listened to the snippets of music, in the other condition (alternate drawing) they took turns. This greatly influenced the nature of the drawings elicited by the task. Drawings in the alternate condition were much more likely to be what Healey and colleagues called figurative (see Figure 7) – the drawings depicting things associated with the music (e.g., iconic representations of a churchyard for funeral music), whereas those in the concurrent condition were abstract (see Figure 7) – lines and blobs tracking the changing pitch, rhythm or stress in the music (see also Healey et al., 2007). They argued that the emergence of more abstract representations, similar in some respects to those from Garrod et al. (2007), depended on having more tightly coupled production and feedback.

Figure 7.

Examples of drawings from Healey et al. (2007) reflecting either Figurative (left) or Abstract (right) properties of the music being communicated.

Diffusion of novel communication systems within and across generations of users

As we pointed out in the introduction, an influential approach to studying the evolution of natural languages uses computer simulations of interactions between artificial agents (Cangelosi and Parisi, 2002). Some simulations assume an evolutionary principle analogous to iterated learning in which the language is transmitted vertically down generations of speakers (Kirby, 2002). Others attribute evolution to processes of social coordination within communities of communicators (Steels, 2003). This contrast is also reflected in ES. For example, the iterated learning studies by Kirby et al. (2008) and Cornish (2010), discussed in Section “Emergence of Compositionality,” investigate the vertical transmission of artificial languages from generation to generation. By contrast, a number of studies in ES investigate the evolution of communication systems through interactions within the same generation of communicators (e.g., Galantucci, 2005; de Ruiter et al., 2007; Healey et al., 2007; Selten and Warglien, 2007; Scott-Phillips et al., 2009). How is this reflected in the results from the two kinds of experiment?

First, as we discussed above, both arrangements, vertical transmission, and horizontal, can lead to the emergence of compositional communication systems. But do both arrangements lead to the emergence of simplified symbolic signs as illustrated in the study by Garrod et al. (2007)? To date it seems that only when there is horizontal transmission (and interaction) is there systematic simplification of the signs. Garrod et al. (2010) carried out a direct comparison of vertical and horizontal transmission using a version of their graphical communication task. In one condition a pair of participants repeatedly communicated a set of concepts, as in Garrod et al. (2007). In a matched condition, the first drawing from the interacting pair was used to seed an iterated learning chain analogous to the chains in Kirby et al. (2008). So the first member of the chain saw and matched the first set of drawings from the interacting pair and then went on to communicate the same set of items to be matched by the second member of chain. This continued for five generations of players. An example of the drawings produced for “parliament” is shown in Figure 8. Whereas the drawings increasingly become graphically simplified in the horizontal transmission (interactive) condition, there is no evidence of such simplification for the vertical (iterated learning) condition. However, it is still not clear whether the simplification (and associated symbolization) may also depend on interaction and feedback available in the horizontal but not vertical transmission conditions. This requires further investigation.

Figure 8.

Drawing refinement and convergence for the concept “Parliament” (A) across six games between an interacting pair and (B) across six generations in a diffusion chain. Participant numbers are given in bold on the top right of each drawing. (From Garrod et al., 2010).

Another difference between the iterated learning studies by Kirby and colleagues and the graphical communication studies by Garrod, Healey and colleagues relates to the number of participants over which the communication systems diffuse. Whereas an iterated learning experiment may involve a chain of 10 players, the standard graphical communication task is dyadic with only two players. To extend the graphical communication paradigm Fay et al. (2010) devised a community version of the task. Four 8-person laboratory communities, or microsocieties, were created via the one-to-one interactions of partners drawn from the same pool. Participants played six consecutive games with a partner. They then switched partners and played a further six games with a new partner, and continued to do so until they had interacted with each of the other community members. The Community condition was contrasted with an Isolated Pair condition, in which participants interacted with the same partner over the same number of games. Figure 9 illustrates the global and local evolution of the sign representing “Brad Pitt” within a single Community and a corresponding number of Isolated Pairs.

Figure 9.

Drawing refinement and convergence for the concept “Brad Pitt” among a Community of interlocutors and between Isolated Pairs at Round 1 and Round 7. Participant numbers are given in bold at the top right of each drawing. (From Fay et al., 2010).

Whereas each isolated pair converges on different simplified drawings, the community as a whole converges on a single simplified (and symbolic) representation of Brad Pitt. So this study demonstrates how communities of interacting pairs will globally converge on a simplified and symbolic communication system. It also turns out that the system they converge on seems better suited for other members of the population than the equivalent systems created by isolate pairs. Fay et al. (2008) had other participants learn to associate the drawings with their concepts produced either by a community or isolated pairs. Once learnt, the community generated drawings could be matched more accurately and quickly with their associated concepts than those produced by isolated pairs. This suggests that communities evolve more effective graphical signs than do isolated pairs.

Role of social dynamics during the development of a communication system

Human languages often change when two or more groups which share a common language but differ in other aspects of their social identity interact with one another. For example, when a group of people wishes to express a difference in social stance with respect to the larger community to which the group belongs, the group tends to adopt a peculiar way of articulating the language they share with the community, so to clearly identify themselves (Labov, 1972). Can we learn something about the social dynamics which govern language change by studying them in the laboratory? To answer this question, Roberts asked people to use newly learned artificial languages in order to negotiate a series of transactions in a group version of a semiotic matching game (Roberts, 2008, 2010). He then observed how the languages used by the players changed depending on two social factors: frequency of interaction (how often people played with a given partner) and competition (whether or not there were competing teams in the game). The main result of the study was that, with frequent interactions, players became able to identify one another on the bases of subtle linguistic cues. However, this led to linguistic divergence among players only in the competitive condition. Considering that the game lasted only a few tens of minutes, this study suggests that, when human interactions are both conflictive and frequent, linguistic divergence can occur at a very fast pace.

The emergence of communication

One of the opportunities offered by ES is that of observing the very emergence of communication, that is, how people bootstrap a communication system without the aid of a previous one.

Semiotic coordination games are particularly suited for studying the emergence of communication as in these games people face the most severe semiotic challenge. Indeed, studies performed with semiotic coordination games have identified a few key elements for the successful bootstrapping of communication (Galantucci, 2005; Scott-Phillips et al., 2009; De Ruiter et al., 2010). At the same time, people engaged in semiotic coordination games sometimes fail at developing even a minimal communication system, providing further insights into the recipe for establishing successful communication systems (Galantucci and Steels, 2008; Galantucci, 2009). In this section we present the main findings of these studies. As it will become apparent in a moment, some of these findings originate from experimental manipulations whereas others are qualitative observations. We included the latter in this review because very little is known about the early stages of emerging communication systems in humans and, at this juncture, qualitative observations can provide useful insights for future systematic scientific inquires.

Insights from successes

Galantucci (2005) provided a qualitative analysis of the process that lead players in a semiotic coordination game to bootstrap communication. Two main processes were identified: learning by observing/using, and naming procedures. The former process referrers to the fact that, through repeated iterations of the game, some players in the study begin to identify contingencies between the partner's location (or moves) in the game and the signing activity produced by the partner (learning by observing). Once such contingencies were identified, players often investigated the exact meaning of their partner's signing activity by using the same signing activity themselves and monitoring how the partner reacted to that (learning by using). When used simultaneously by two adaptive players, these complementary learning processes provide a powerful mechanism to bootstrap a communication system. This mechanism is similar to the mechanism of interactive input–output alignment described by Garrod and colleagues in the context of conversations using natural language (Garrod and Anderson, 1987; Garrod and Pickering, 2004). In other words, the bootstrapping of communication might rely on a mechanism that is not different in kind from the mechanism that facilitates convergence on the fine details of how to use pre-established forms of communication. A different type of process to bootstrap communication – naming procedures – relied more directly on explicit negotiations. These negotiations, which consisted of communicative behaviors not oriented toward the immediate goal of coordinating moves in the game but rather toward the goal of future mutual understanding, will be described in Section “Insights from Failures.”

Another insight into the emergence of communication provided by the study performed by Galantucci (2005) was that, in order to achieve coordination, people did not depend exclusively on the use of communicative signs, even when all the needed signs were available. Galantucci observed that, when the circumstances of his game allowed it, coordination was achieved by integrating the information provided by publicly broadcasted signs with other kinds of task-relevant information. For example, when the game environment comprised only three possible longitudes and players discovered through the use of signs that their agents were at the same latitude, they rarely used further signs to coordinate their moves. Since their goal was that of bringing the agents in the same game location, they simply moved toward the central longitude (If an agent was already in the central location, the player would not move it.). In other words, the behavior of the players reflected the efficient integration of (a) information contained in the signs (obtained through the communication medium and explicitly shared by the players), (b) information about the current location of the controlled agent (obtained privately and not shared by the players), and (c) information about the environment layout (obtained privately and implicitly shared by the players). Such efficient integration is typical of natural language use (Tanenhaus et al., 1995; Goodwin, 2000; Clark, 2005).

Using a different procedure, Scott-Phillips et al. (2009) focused on a very basic question concerning the emergence of communication: how do actions assume a communicative value? They addressed this question using a coordination semiotic game in which players had to move agents in a game environment to achieve coordination but had no other means to communicate than the very moves their agents made in the game. This forced them to find ways to signal that some of the moves that their agents made did not have the intent of bringing the agents to a specific point of the game environment but rather that of communicating something to the partner. In other words, players had to find ways to “signal signalhood.” The challenge proved substantial and five out of 12 pairs failed at the task. The process that led the remaining pairs to succeed was typically composed of two steps. The firsts step was to establish a salient default coordination procedure, which consisted in a shared decision bias for the “right” move to be made (cf. Schelling, 1960). The second step occurred when the default coordination procedure was not viable. This was typically signaled by making a series of moves which would have been unnecessarily complex had the default coordination procedure been available. The unexpectedness of these moves prompted the partner to recognize a communicative intent and, over time, these moves came to signify new coordination procedures, enriching the communicative repertoire of the pairs. Scott-Phillips and colleagues demonstrated the fundamental importance of the first step via a simple manipulation. When default coordination procedures were made more difficult to establish, failures in the game occurred about twice as frequently.

De Ruiter et al. (2010) focused on the cognitive infrastructures underlying successful human communication. In particular, they investigated the complementary emergence of behaviors that imply the intent to communicate to an audience (recipient design) and of behaviors that imply intention recognition. De Ruiter et al. (2010) used a coordination semiotic game, the tacit communication game (TCG), in which a player had to perform two tasks simultaneously. The first task was that of moving and rotating an object on a grid in order to place it in a target location with a specific orientation. The second task was that of using the moves on the board to communicate to the partner the target location and orientation for the object s/he controlled (see Figure 10).

Figure 10.

Sequence of events in the Tacit Communication Game developed by de Ruiter and colleagues.

1. Players view their shapes (1500 ms)

2. Player(s) see the goal configuration

3. The sender starts his movement sequence by pressing the start button

4. The sender has 5000 ms to move the shape around on the game board

5. The receiver begins movements from center tile by pressing the start button (the sender's shape remains on screen in position they left it)

6. The receiver has 5000 ms to move her shape around in the game board

7. Players receive feedback indicating whether they were correct (green box) or incorrect (red box) in matching the goal configuration (both the sender and the receiver received this feedback, indicating whether they were successful as a team in reproducing the goal configuration)

   (Adapted from de Ruiter et al., 2010).

Although the communicative challenge was not trivial as the objects, the target locations, and the target orientations could all be different, people were able to perform the task successfully. De Ruiter et al., (2010) manipulated the conditions in which the game was played with the intent of directly investigating the mechanisms that led to success. The results of their experiments support three main conclusions. First, feedback is important for the emergence of successful communication, a result that confirms the conclusions of a number of referential semiotic games (e.g., Garrod et al., 2007; Healey et al., 2007). Second, the difficulty of a communicative act is reflected in the planning times of both players involved in it, indicating that the task involves both recipient design (i.e., communicating by taking into account the interlocutor's stance) and intention recognition. Third, there is no trade-off between the planning time in senders and receivers, indicating that when communicative acts are difficult, the difficulty of intention recognition is not mitigated by sophisticated forms of recipient design (and vice versa).

Intriguingly, behavioral procedures for communication similar to the one observed in the two studies above were also observed in semiotic coordination games in which players had at their disposal a dedicated means of contact (a small digitizing pad), which was completely independent from the actions of the agents in the game (Galantucci and Steels, 2008; Galantucci, 2009). These procedures, which are related to the naming procedures mentioned above, are illustrated in the following section.

Insights from failures

Sometimes players engaged in semiotic coordination games fail at developing even a minimal communication system (Galantucci, 2005; Scott-Phillips et al., 2009). Such failures provide useful information about the core ingredients for the emergence of communication. As we have seen in the previous section, Scott-Phillips et al. (2009) used a simple experimental manipulation to demonstrate that failure is more likely when players are not afforded the possibility of establishing a salient default coordination procedure. Direct comparisons between the behaviors of failing and successful players (Galantucci and Steels, 2008; Galantucci, 2009)suggested two further conclusions. The first one is that the failures are often due to communicative egocentrism. Such egocentrism, which has been noted before in spoken conversation (Keysar, 2007), seems to be much heightened in the presence of a severe semiotic challenge. For example, some players did not use the digitizing pad they had at their disposal to communicate for as long as two consecutive hours, often while the partner tried repeatedly to initiate some form of communication. These players sometime made use of the signs generated by the partner, demonstrating an understanding of the basic dynamics of the game. However, the idea of reciprocating the communicative acts initiated by the partner was not obvious to them, suggesting a severe egocentric limitation. This conclusion is consistent with the behavior of other players who were successful at the game but developed signs which had different meaning depending on something that they privately controlled – the trajectory of the stylus on the digitizing pad – but which was not publicly perceivable. For example, some players drew a vertical line on the pad from top to bottom to indicate “going down” and from bottom to top to indicate “going up.” Given the constraints of the communication medium, the two drawings led to identical tracings on the screen seen by the partner (as well as by the drawer). However, the drawings felt different to the tracer's hand and were thus used as different signs.

The second conclusion is that explicit negotiations does not rescue failing players. In the semiotic coordination game developed by Galantucci and colleagues (Galantucci et al., 2003; Galantucci, 2005), players controlled agents which had a simplified human body, with clearly distinguishable front and back sides. When the agents were in the same room of the game environment, players could see both of them and could use the orientation of the agent's front side, as well as the agent's location and movements in the room, for communicative purposes. For example, players could move the agent they controlled close to a door in the room, make it face the door, and move it in an oscillating manner near it. Behaviors of this kind were easily interpreted by the partner as “pointing” at the door and allowed players to establish relations between the meaning of the signals produced with the digitizing pad and the objects pointed to (this was referred to above as a “naming procedure”). Galantucci (2009) found that, overall, explicit negotiations of this kind were not necessarily beneficial for success in the game, a finding which parallels a finding in experimental pragmatics (Garrod and Anderson, 1987). Indeed, unless players had a keen ability to coordinate their joint attention and actions – in which case they typically engaged in explicit negotiations sparingly and greatly benefited from them – they were easily led to utter confusion by such negotiations (Galantucci and Steels, 2008; Galantucci, 2009). In other words, explicit negotiations might help people who are already capable of communicating to fix certain aspects of their communication system but they are not a very efficient option for the bootstrapping of communication (Galantucci and Steels, 2008). This conclusion is consistent with the conclusion of the study by Scott-Phillips et al. (2009) that succeeding in a semiotic coordination game is much harder when communication cannot be bootstrapped implicitly.

Implications for Cognitive Neuroscience

The ES studies reviewed above use only behavioral manipulations and measures. However, ES is also amenable to neuroscientific investigation. For instance, Noordzij et al. (2009) report an fMRI study of TCG players in which they manipulated the communicative intent of a sender's movements. They compared standard communicative trials with trials in which the sender was instructed to copy the same movements without communicative intent. They also varied the difficulty of the communicative trials. Their main finding was that communicative difficulty modulated BOLD responses in the posterior STS, a region associated with conceptual mind-reading activities (Frith and Frith, 2006). Furthermore, this posterior STS BOLD modulation occurred both for senders generating the signals and receivers trying to decode those signals.

Noordzij et al. (2009) used an event related fMRI design. Another possible approach is to investigate how communicators align their neural activity over time as a consequence of communication. Stephens et al. (2010) did this for speakers producing and listeners comprehending a spontaneous narrative. To establish alignment they correlated cortical BOLD signal changes between speakers and listeners over the whole course of the narrative. Interestingly, they found evidence for aligned neural activation in many cortical areas at different lags; on occasion the speaker's activity preceded that of the listener, but there were occasions when the reverse was true. Furthermore, those listeners’ whose neural activity aligned well with that of the speaker showed superior comprehension of the story. In a similar vein, Schippers et al. (2010) developed a between-brain Grainger Causality Measure to establish the causal relationship between activity in the brain of someone performing manual gestures and that of someone identifying the gestures. They demonstrated a strong causal influence between activity in the gesturer's putative Mirror Neuron System (Keysers and Gazzola, 2009) and that of the person identifying the gesture, with this relationship only being clearly apparent during periods of communication as opposed to simple observation of the gesture. These methods of correlating neural activity between producers and receivers seem well suited for investigating basic communicative processes of the kind studied in ES.

Finally, ES paradigms could be used with populations suffering from certain brain lesions to establish the nature of any associated deficit. For example, they might prove useful in characterizing general communicative deficits in aphasic populations, possibly leading to therapeutic applications.

Conclusion

In this review, we (a) situated ES in its conceptual context, (b) illustrated the main varieties of studies thus far conducted by experimental semioticians, (c) illustrated three main themes of investigation which have emerged within this line of research, and (d) considered its implications for cognitive neuroscience. Here we would like to conclude with two general remarks concerning ES. The first remark concerns the scope of its scientific contributions. As we have seen in Section “Main Themes of Investigation in ES,” experimental semioticians have uncovered a number of core principles of human communication. We believe that such principles have a broad scope and could be relevant for the study of all forms of human communication. For example, they can contribute to understanding the design of natural languages (spoken as well as signed). Indeed, as we have seen in Sections “Early Emergence of Linguistic Structure” and “Social Manipulations in ES,” the novel communication systems observed by experimental semioticians share important similarities with natural languages, quickly adopting abstract symbols (Garrod et al., 2007; Healey et al., 2007), and exhibiting some of the core features of a linguistic design (Selten and Warglien, 2007; Kirby et al., 2008; Galantucci et al., 2010; Theisen et al., 2010). Should we interpret this as clear evidence that the core design of natural language can be directly explained by general principles of human communication and that ES is an ideal methodology for uncovering such principles?

A positive answer to this question depends on two assumptions. The first one is that the development of novel communication systems does not differ in crucial ways when it occurs in the laboratory rather than in a more natural context. The status of this assumption is yet to be determined although we believe that the rapid growth of ES will soon lead to such a determination. In any case, the eventual falsification of this assumption would not automatically lead to the dismissal of ES because, as we mentioned in Section “Main Varieties of Studies in ES,” experimental semioticians have ample margins to refine their methodology in order to deal with the problem.

The second assumption is that experimental semioticians are successful in preventing pre-existing communication systems from influencing the results of their studies. Participants in the studies reviewed here were all adults with vast knowledge of their mother tongue as well as of a number of other well-established communication systems (e.g., road signs). Such knowledge might leak into the experiments designed by the experimental semioticians, making it difficult for them to isolate the true sources of the communication systems they observe. Considering that players often fail at games which would be easily mastered by using pre-existing communication systems (cf. Insights from Failures), we believe that such leakage is fairly minimal. However, we will not defend that claim here.

Rather, we suggest that a possible way to address the potential confound is to carry out ES studies with pre-linguistic children or animals. This is a considerable scientific challenge, which to our knowledge has not yet been undertaken. We hope that this review will stimulate researchers to tackle it soon, testing along the way whether ES can be a successful methodology for studying social communication in its most general forms. And this leads us to our second and final remark.

Up until now, most of ES has been conducted at a behavioral level of analysis. However, as we argued above ES is also amenable to neuroscientific investigation and this line of research seems particularly promising to us because studying neural correlates of cognitive activities related to social interactions is most likely to succeed in the context of tasks that are intrinsically social (see Sebanz et al., 2006; Galantucci and Sebanz, 2009 for similar arguments in relation to behavioral correlates). It is our hope that this review in a journal primarily focused on neuroscience will stimulate more research in this direction.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.