Web-based experiments for the study of collective social dynamics in cultural markets

Abstract

Social scientists are often interested in understanding how the dynamics of social systems are driven by the behavior of individuals that make up those systems. However, this process is hindered by the difficulty of experimentally studying how individual behavioral tendencies lead to collective social dynamics in large groups of people interacting over time. In this paper we investigate the role of social influence, a process well studied at the individual level, on the puzzling nature of success for cultural products such as books, movies, and music. Using a “multiple-worlds” experimental design we are able to isolate the causal effect of an individual level mechanism on collective social outcomes. We employ this design in a web-based experiment in which 2,930 participants listened to, rated, and download 48 songs by up-and-coming bands. Surprisingly, despite relatively large differences in the demographics, behavior, and preferences of participants, the experimental results at both the individual and collective level were similar to those found in Salganik, Dodds, and Watts (2006). Further, by comparing results from two distinct pools of participants we are able to gain new insights into the role of individual behavior on collective outcomes. We conclude with a discussion of the strengths and weaknesses of web-based experiments to address questions of collective social dynamics.

1. Introduction

Can you study an army in a laboratory? In 1969 the sociologist Morris Zeldich posed this provocative question about the study of group behavior. At the time Zeldich concluded that because logistical difficulties prevented researchers from studying large groups literally “in a lab,” the more realistic way to learn about these groups was by studying the smaller-scale process at work within them (Zelditch, 1969). Almost 40 years later, however, it may be time to revisit Zelditch’s question with a fresh perspective. In particular, the vast increase in computing power over the past few decades, and the almost limitless pool of participants now available via the Internet have made it possible—although by no means trivial—to design and conduct laboratory-style experiments involving thousands, or even millions, of participants. Moving beyond the classic study of “small groups” (Harrington & Fine, 2000), we can now begin the study of “large groups” (Hedström, 2006).

But first it might be asked, why the ability to study large groups experimentally is needed when we already have the ability to study individuals experimentally? An example that combines the individual-level process of anchoring and adjustment (Tversky & Kahneman, 1974) with the collective dynamics of auctions proves instructive. Individuals have been found to anchor their valuation of objects based on irrelevant (Ariely, Lowenstein, & Prelec, 2003; Simonson & Drolet, 2004) and even false information (Northcraft & Neale, 1987). For example, real estate agents’ valuations of homes has been shown to be affected by false listing prices (Northcraft & Neale, 1987). Therefore, because of this psychological mechanism, one might predict that auctions with higher starting prices would have higher final sale prices because auction participants, like the real-estate agents, would anchor on the higher starting price. Yet, in real auctions exactly the opposite can occur: lower starting prices can lead to higher final prices (Ku, Galinsky, & Murnighan, 2006). In other words, even thought the individual psychology of anchoring has been studied extensively, it offers us the wrong prediction about the collective social dynamics of auctions.

More generally than auctions, it seems to be difficult to predict the behavior of systems with many interacting components, even with a reasonable understanding of these lower-level components. That is, just as it is hard to reduce biology to the behavior of molecules (Hartwell, Hopfield, Leibler, & Murray, 1999), it is hard to reduce group behavior to individual psychology (Barton, 1968); in the words of Anderson (1972), “more is different.” This observation does not imply that individual behavioral tendencies tell us nothing about group behavior, as has been argued by proponents of the “collective mind” and the “madness of crowds.” However, it does suggest that models of collective behavior that are based solely on the behavior of a “representative agent” are seriously deficient (Kirman, 1992). The challenge for studying collective behavior is therefore in understanding how the actions and interactions of individuals aggregate to produce collective patterns (Allport, 1924; J. S. Coleman, 1990; Hedström, 2005; Schelling, 1978).

Much of the formal work connecting individual behavior and group dynamics has relied on mathematical and agent-based models (Arthur, 1994; Axelrod, 1997; Banerjee, 1992; Bikhchandani, Hirshleifer, & Welch, 1992; Goldstone, Ashpole, & Roberts, 2005; Macy & Willer, 2002; Watts, 2002), but these approaches are invariably subject to the critique that they are based on overly simplistic representations of human behavior and so are of questionable relevance to real-world social phenomena. In this paper, therefore, we will attempt to address these same issues with experiments involving human subjects making decisions that—although extremely simple—are not so different from decisions they make in their everyday lives. Specifically, we will investigate the effect of social influence, a process well studied at the individual level, on the success of products in cultural markets.

The paper will be organized as follows. First, we will describe the puzzling nature of success and failure in cultural markets, a collective outcome, and then show how the “multiple-worlds” design of Salganik, Dodds, and Watts (2006) allows us to address the role of individual-level social influence on collective outcomes. Unlike most experiments in cognitive science, psychology, and economics that have individuals as the unit of analysis, these experiments have groups of several hundred people as the unit of analysis, and therefore require thousands of participants, running directly into the constraints of the physical laboratory that troubled Zeldich. We were able to circumvent these problems by running our experiment over the Web, allowing us to accommodate many more participants than have been possible in a physical laboratory. After reviewing the results from two previous experiments whose participants were largely American teenagers (Salganik, Dodds, & Watts, 2006), we will present new results with an entirely distinct pool of participants with different demographics, preferences, and behavior (n=2,930). Comparison of these results reveal additional features of the role of individual behavior on collective outcomes that would not be possible by studying only one participant population. The paper concludes with a discussion of the possibilities and limitations of the web-based experimental study of large group behavior.

2. Success and failure in cultural markets

An obvious “stylized fact” about markets for cultural products such as books, movies, TV shows, and music is that success in such markets is tremendously unequal (Caves, 2000; Chung & Cox, 1994; De Vany, 2004; Sorensen, 2007; Vogel, 2004). That is, blockbuster movies like Star Wars and bestselling books like Harry Potter are not merely more successful than average—they are orders of magnitude more successful; so much so that cultural markets are often characterized as “superstar” (Rosen, 1981) or “winner-take-all” (Frank & Cook, 1995) markets. On its own, there is nothing particularly puzzling about this observation: because the cost of reproducing cultural products is generally unrelated to their quality, then as long as some products are distinctly better than others, everyone can enjoy “the best” for essentially the same price as an average product. Thus one ought to expect that success in cultural markets will be highly skewed at the top-end (Rosen, 1981).

What is more puzzling, however, is that if these “superstar” products—The Beatles, Harry Potter, Star Wars—are somehow different from all the other products in ways that are obvious to audiences, one might expect that these same differences would be evident to the producers, publishers, and executives responsible for deciding which projects to support and which to reject. Yet along with inequality of success, a second stylized feature of cultural markets is that they appear to be highly unpredictable, in the sense that attempts to predict hits routinely fail. For example, eight publishers rejected the first Harry Potter book before it went on to sell millions of copies around the world and launch a billion dollar franchise (Lawless, 2005). Executives at the TV network FOX must have bitterly regretted passing on the sit-com Friends, but reveled in their decision to broadcast the mega-hit American Idol after it was rejected by executives at NBC, CBS, and ABC (Carter, 2006). Even some of the most successful cultural products of the 20^th century had these troubles; The Beatles struggled to get a record deal (R. Coleman, 1989) and Star Wars almost never got made (Seabrook, 2000).

Just as there are products that are initially rejected, only to end up wildly successful, there are also products that are expected to have wild success that flop (although these are harder to document because they are rarely written about). One such example is Carly Hennessy, an aspiring pop star who impressed MCA Records so much that they invested $2.2 million dollars in the production and marketing of her first album, Ultimate High. Yet, once the album was released it sold just 378 copies in its first three month (Ordonez, 2002). Stories similar to Hennessy’s are not uncommon; failures greatly outnumber successes. It is estimated that major record labels lose money on 90% of the artist they sign, and similar, but slightly lower, failure rates are estimated for other cultural products (Vogel, 2004).¹ Ethnographic studies of decision making in cultural industries find that, not surprisingly, industry insiders are very aware of this situation (Bielby & Bielby, 1994; Denisoff, 1975; Faulkner & Anderson, 1987; Gitlin, 1983; Hirsch, 1972), and their feelings are nicely summarized by the well-known quip of the screenwriter William Goldman (1983): “nobody knows anything.”

The puzzle we address in this paper, therefore is the following: if hits are different in some way, why do experts have such difficulty identifying these products ahead of time²? Rather than looking for an explanation of this puzzle at the level of the individual—either industry executives or consumers—we will argue that the inequality and unpredictability of success, both group level properties, arise from a process of social influence at the individual level. That is, rather than assuming exogenous and stable preferences, as is common in rational choice models of human behavior and models cultural markets (e.g. Rosen (1981)), our approach is more consistent with a great deal of recent work in experimental psychology suggesting that preferences are neither exogenous nor stable, but are in fact “constructed” by a variety of features of the decision context itself (Bowles, 1998; Payne, Bettman, & Johnson, 1992; Payne, Bettman, & Schkade, 1999; Slovic, 1995). Our explanation, therefore, takes the constructed preferences view of human psychology seriously, and adds to it a social component; that is, we propose that expressed preferences are influenced by the observed actions of others, as well as by psychological features of the decision context like framing, anchoring, and availability (Ariely, 2008).

3. Social influence and cumulative advantage

The interdependence of decision making, and more specifically the processes of social influence and conformity, have been studied extensively by psychologists, economists, and sociologists for at least 70 years (Asch, 1956; J. S. Coleman, Katz, & Menzel, 1957; Crutchfield, 1955; Deutsch & Gerard, 1955; Katz & Lazarsfeld, 1955; Kelley & Thibaut, 1978; Leibenstein, 1950; Luce & Raiffia, 1957; Sherif, 1936; Thibaut & Kelley, 1959). Research in this area has lead to two broad insights that are relevant to social influence in cultural markets. First, in cultural markets there are simply too many products for individuals to consider—for example, in one year alone, about 6,500 new major-label albums are released (Vogel, 2004). Since no one can possibly consider all of these products, a natural heuristic for dealing with this choice overload is to assume that the popularity of products is somehow a signal of their quality, a phenomena sometimes called observation learning (Banerjee, 1992; Bikhchandani et al., 1992; Goldstein & Gigerenzer, 2002; Hedström, 1998). And second, in cultural markets, people may benefit from coordinating their choices with others (Adler, 1985); that is, by listening to, reading, or watching the same things, friends, and even strangers, construct common points of interest around which they can interact, thereby fostering notions of commonality and community. Although differentiating between these two kinds of effects can be difficult in practice (Cialdini & Goldstein, 2004; Deutsch & Gerard, 1955; Moretti, 2008), for the purpose of understanding collective dynamics, it may be sufficient to note that both correspond to an individual-level tendency to imitate the behavior of others. From this perspective, popular products will tend to become more popular, leading to “cumulative advantage” or “rich-gets-richer” dynamics, as recent experiments have demonstrated (Hanson & Putler, 1996; Salganik et al., 2006)

Based on simple assumptions about these cumulative advantage processes, a range of mathematical and simulation models have been proposed that produce (a) highly skewed distributions of outcomes (Barabási & Albert, 1999; de Solla Price, 1976; DiPrete & Eirich, 2006; Simon, 1955; Yule, 1925); and (b) unpredictability due to path dependence effects in which initially indistinguishable states can “lock in” to very different equilibria (Arthur, 1994). Results from these models suggest that the puzzling nature of success in cultural markets could follow naturally from the process of social influence at the individual level. Testing this claim empirically, however, has proven more difficult. One difficulty is the sheer proliferation of such models, each of which makes its own assumptions, and yields somewhat different predictions about collective behavior (Adler, 1985; Arthur, 1994; Banerjee, 1992; Bentley, Lipo, Herzog, & Hahn, 2007; Bikhchandani et al., 1992; Chung & Cox, 1994; De Vany & Lee, 2001; De Vany & Walls, 1996; Dodds & Watts, 2004; Granovetter, 1978; Watts, 2002). Further, and more fundamentally, even with the tremendously accurate and fine-grained sales data that is collected by market research firms, it is hard to convincingly measure the quality of products and therefore hard to convincingly argue that some other outcomes could have happened.

Imagine, for example, trying to convince a Harry Potter-obsessed friend that the book’s success was the results of a cumulative advantage process and that the book could have just as easily been a flop (as was predicted by the eight publishers who passed on it). That fan could easily counter that the success of Harry Potter had nothing to do with luck, but stemmed directly from its attributes, which although not what experts in children’s book publishing had anticipated, must have been “what people wanted” (Tucker, 1999).³ One hypothetical way to resolve this disagreement might be to “rewind” the world to the day that Harry Potter was first released, and “rerun history” a number of times: if Harry Potter is popular in all of these versions of the world, that would be strong support for the claim that its success stems from its intrinsic attributes; but if it became popular in one of these worlds and not the others, then we could suspect that its success was determined in part by chance. In other words, in order to understand the role of chance, we need to observe multiple realizations of the same process. Clearly no such experiment could ever be implemented in existing markets, such as the one that produced Harry Potter. However, we shall argue that it is possible to implement this thought experiment, albeit in a highly simplified form, in an artificial cultural market that we have constructed.

4. Experimental design

The basic framework for our experiment was an artificial cultural “market” created in the form of a website (http://www.musiclab.columbia.edu) where participants could listen to, rate, and download 48 songs by unknown bands.⁴ Unbeknownst to the participants, we used the “multiple-worlds” experimental design⁵ presented in Figure 1. Upon arrival to the website participants were randomly assigned into either the independent condition, where they had no information about the behavior of others, or the social influence condition and then a specific “world” where they had information about the behavior of those in their world, but not those in other the worlds. Because all the worlds started with the same initial conditions, had indistinguishable participants, and the same 48 songs, they represent multiple parallel realization of the same process.

Figure 1.

Schematic of the experiment designed used in these experiments. This design allows researchers to observe the effects of the individual process of social influence on collective outcomes.

The participants, although not aware of these multiple worlds, were told that they were participating in a study of music tastes and that in exchange for participating they would be offered a chance to download new music by up-and-coming artists.⁶ After providing their informed consent,⁷ participants completed a brief survey that collected simple demographic information such as age, gender, country of residence, etc. Next, participants were sent to a page of instructions and then, finally, were presented with a song menu (Figure 2), similar to the layout of other online music downloading sites. For participants in the social influence worlds, the band names and song names were accompanied by the number of previous downloads in their world; while for participants in the independent condition no such information was provided. Participants could click on a song at which point it began to play. While the song was playing participants were asked to rate the song on a scale of 1 star (“I hate it”) to 5 stars (“I love it”). After rating the song, participants were offered the chance to download it, and then were taken back to the song menu and were able to listen to other songs. The download counts presented to the participants were updated accurately and in real-time.⁸ Because of the design of our website participants could not download songs without listening to them, but they could listen to, rate, and download as many songs as they wished.

Figure 2.

Screenshots of the song selection screen from experiment 1 (A) and experiments 2 and 3 (B). Note that in experiment 1 the placement of songs was random, but that in experiments 2 and 3 the songs were sorted by popularity. Screenshots from the independent condition were identical, except the songs were always ordered randomly and the download counts were not displayed.

5. Previous results

Using the same experimental setup, Salganik, Dodds, and Watts (2006) conducted two successive experiments that varied in the amount of social influence to which individual decision makers were exposed. These experiments took place from October 7, 2004 to December 15, 2004 (69 days) and December 15, 2004 to March 8, 2005 (83 days), and collectively involved a total of 14,341 participants. The participants were mainly recruited from the now defunct, teen-interest website http://www.bolt.com using a combination of text ads and banner ads Consistent with the site’s demographics, most participants were teenagers living in the United States (Table 1).⁹ In each experiment, participants arriving to our website were randomly assigned to one of eight social influence worlds or the one independent world, such that the independent world received twice as many participants as the social influence worlds (the reason for this difference will become clear when we describe our measure of unpredictability). Given a total of 14,341 participants over the two experiments, therefore, each social influence world had about 700 participants, while each independent world had about 1,400.

Table 1.

Descriptive statistics of participants in experiments 1, 2, and 3.

Category	Experiment (n=7,149) (% of participants)	Experiment 2 (n=7,192) (% of participants)	Experiment 3 (n=2,930) (% of participants)
Female	36.4	73.9	38.0
Broadband connection	74.1	69.0	90.6
Has downloaded music from other sites	60.4	62.4	69.3
Country of Residence
United States	79.8	81.8	68.4
Canada	4.5	4.4	6.3
United Kingdom	4.4	4.7	6.6
Other	11.0	9.1	18.7
Age
17 and younger	11.5	16.0	1.5
18 to 24	27.8	34.9	5.7
25 to 34	38.5	39.2	29.8
35 and older	22.3	9.9	63.1

In experiment 1, songs were presented in a 16×3 grid not sorted by popularity, while in experiment 2, the songs were presented in a single column format sorted by popularity (Figure 2). In the independent condition of both experiments, the songs were presented in the same format, but without any information about popularity and in a random order. Thus, by comparing the results within each experiment we could observe the effects of individual social influence on collective outcomes and by comparing across experiments we can observe the effect of increasing the “strength” of this individual-level process (i.e. by making the social information more salient to the users).

Before considering our main question about inequality and unpredictability of success, we first measure whether the information about the popularity of the songs affected participants’ behavior at all, and whether this effect was larger in experiment 2, where the information was made more salient. To construct such a measure, we note that at the time that each subject participated, every song in his or her world had a specific market share and market rank (for example, the song with the highest market share had a market rank of 1). Figure 3 plots the probability that a participant listened to a song of a given rank (independent of which song occupied that rank at that time) in experiments 1 and 2. Figure 3A shows that in experiment 1, when the songs were not sorted by popularity, participants in the social influence worlds had about a 20% chance of listening to the most popular song, but only about a 7% chance of listening to a song of average popularity (solid circles). In the independent world (open circles), the probability of listening was independent of market rank, which is as expected given our design. Figure 3B shows the same results from experiment 2 and reveals that, as expected, sorting the songs by popularity caused participants to be more affected by market rank. For example, in experiment 2, participants had almost a 50% chance of listening to the most popular song, but only about a 5% chance of listening to a song of middle market rank.¹⁰

Figure 3.

Probability of listening to a song of a given market rank. The pattern shows that participants in experiment 2 were more likely to listen to more popular songs.

These results therefore confirm that the popularity of the songs affected participants choices and generally lead them to listen to the more popular songs—a result that is consistent with the large literature on social influence and conformity. Now, we turn our attention to measuring the consequences of this individual level social influence on collective outcomes, particularly the inequality and unpredictability of success. To begin with, we define the success of a song based on its market share of downloads,

$$m_i=\frac{d_i}{\sum d}$$

We then measure the inequality of success in terms of the Gini coefficient, a common measure of inequality that is defined as

$$G=\frac{\frac{1}{S^2}\sum _{i=1}^S\sum _{j=1}^S|m_i-m_j|}{2•\frac{\sum m}{S}}$$

where m_i is the market share of song i and S is the number of songs (Allison, 1978; Coulter, 1989).¹¹ The Gini coefficient can be thought of as the expected difference in market share between two randomly chosen songs that has been scaled to fall between 0 (complete equality) and 1 (complete inequality).

Figure 4A shows that in experiment 1 success was more unequal in the social influence worlds than the independent world. To assess whether differences this large could have occurred by chance (i.e., to assess statistical significance), we used a modified bootstrap procedure (Efron & Tibshirani, 1993); that is, we randomly re-sampled the participants (with replacement) from the independent condition to produce 1,000 replicate independent conditions that were similar in size to the social influence worlds and calculated the Gini coefficient for each of these replicates. The value presented in Figure 4A is the mean Gini coefficient of these replicates, and the interval is the range within which 95% of these replicate values fell. The probability that a randomly chosen independent world replicate was more unequal than a randomly chosen social influence world was small (p < 0.01), indicating that the observed differences were statistically significant. Figure 4B shows that a similar pattern was observed in experiment 2 with success in the social influence worlds being more unequal than in the independent world (p < 0.01).

Figure 4.

Inequality of success in experiments 1 and 2 as measured by the Gini coefficient. In both experiments, success in the social influence worlds was more unequal than success in the independent world. Also, success in the social influence worlds in experiment 2 was more unequal than success in the social influence worlds in experiment 1.

Further, the between-experiment comparison shows that in experiment 1, when the songs were randomly ordered, the inequality in the social influence worlds ranged from 0.28 to 0.42, and that in experiment 2, when the songs were sorted by popularity, it ranged from 0.45 to 0.56. In other words, every social influence world in experiment 2 was more unequal than the most unequal in experiment 1.¹² This difference was not only statistically significant (p < 0.01) but also substantively large—by comparison, the magnitude of this difference is similar to the difference in income inequality between Western European countries such as France, Spain, and the Netherlands (G ~ 0.34) and developing countries such as Nigeria, Peru, and Venezuela (G ~ 0.50) (UNDP, 2004). Finally, Figures 4C and 4D plot the dynamics of the Gini coefficients as the experiments progressed and show that the approach to steady-state was qualitatively similar in both experiments, and that we would not anticipate very different results, had the experiment continued.¹³

Next we turn our attention to the unpredictability of success. Recall that the cumulative advantage models predict that we will observe different outcomes across realizations of exactly the same process (Arthur, 1994). To quantify this, we define the unpredictability of specific song, u_i, to be the average difference in market share between two randomly chosen success outcomes. That is,

$$u_i=\frac{\sum _{i=1}^R\sum _{j=i+1}^R|m_{ij}-m_{ik}|}{\left(\begin{array}{c}\hfill R\hfill \\ \hfill 2\hfill \end{array}\right)}$$

where m_ij is the market share of song i in world j and R is the number of realizations. If a song earns the same market share in all realizations then its unpredictability is defined to be zero. Note that zero unpredictability by this measure does not mean that in practice it would be easy to predict the song’s success, but since the song has the same outcome in all realizations, precise prediction cannot be ruled out in principle. To the extent that a song experiences different levels of success in different realizations, however, these differences represent inherent unpredictability that cannot be eliminated any more than one can eliminate the uncertainty in the roll of a die. The unpredictability of an experimental condition was defined to be the average unpredictability of the songs in that condition, $U=\raisebox{1ex}{$\sum _S{u}_i$}\!\left/ \!\raisebox{-1ex}{$S$}\right.$.

Recall, however, that there was only one realization of an independent world and so it seems impossible to calculate unpredictability, which was defined based on differences across multiple realizations of the same process. It was for this reason that the independent condition had twice as many participants as each social influence world. In order to calculate the unpredictability for the independent condition we randomly split the independent world of about 1,400 participants into two independent worlds of about 700 participants each. The unpredictability in the independent condition was then calculated by comparing outcomes in these split worlds, and this procedure was repeated 1,000 times.

Figure 5A and 5B show that the unpredictability of success was greater in the social influence condition than the independent condition. To calculate a measure of statistical significance we compared the distribution of replicate values from the independent condition to the distribution of calculated unpredictability values for the 28 (8×7/2) possible pairs of social influence worlds (the 95% intervals for these distributions are shown on the figures). The probability that the unpredictability of a randomly chosen pair of social influence worlds was greater than the unpredictability of a random split of the independent condition was small (p < 0.01) in both experiment 1 and 2. Thus, we conclude the difference in unpredictability across the conditions is statistically significant. Moreover, comparing the values of our unpredictability measure between experiments 1 and 2, we also find that increasing the amount of social influence increased the unpredictability of success by about 50%. We note, however, that although a difference this large is unlikely, it is not statistically significant by common standards (p ≈ 0.06). Finally, Figures 5C and 5D again show that we would not have likely seen different results if the experiments had continued and that the approach to steady-state was qualitatively similar in both experiments.

Figure 5.

Unpredictability of success in experiments 1 and 2. In both experiments, success in the social influence worlds was more unpredictable than success in the independent world. Also, success in the social influence worlds in experiment 2 was more unpredictable than success in the social influence worlds in experiment 1.

One feature that our measure of unpredictability obscures is that the success of the songs was not completely random, but rather, that some songs consistently did better than others. An advantage of our experimental design is that it allows us to assess whether that might be because some of the songs were more appealing than others. In general, the “quality” of cultural products is notoriously difficult—some might say impossible—to measure objectively (Becker, 1982; Bourdieu, 1984; Cutting, 2003; DiMaggio, 1987; Gans, 1974); thus we instead refer to the “appeal” of a song, which we define in terms of its popularity in the independent condition. We note that what we are calling “appeal” is not necessarily the same as quality. Nevertheless, because the behavior of users in the independent condition is, by definition, unaffected by the behavior of others, it provides a natural measure of pre-existing population preferences--something that has also proven difficult to measure convincingly in previous studies of cultural markets (Hamlen, 1991; Krueger, 2005). More precisely, we measure the appeal of the songs to our pool of participants by the market share of downloads in the independent world:

$$a_i=\frac{d_i}{\sum d}$$

where d_i is the number of downloads for song i in the independent condition (Salganik et al., 2006; Salganik & Watts, 2008). Figure 6 shows that higher appeal songs tended to do better, but there is substantial scatter in the relationship, particularly for the highest appeal songs. In other words, outcomes were more unpredictable for higher appeal songs than for lower appeal songs, suggesting in turn that it is easier to predict failure than to predict success.

Figure 6.

The relationship between appeal, as measured by market share in the independent condition, and success, as measured by market share in the social influence worlds. The higher appeal songs do better on average.

Further, if one were to average the success across the eight social influence worlds, this average success is strongly associated with the underlying appeal; however, it is non-linear in the sense that small changes in appeal at the top of the distribution are associated with larger changes in average success, especially in experiment 2.¹⁴ Thus, one way to consider these results is that the outcome of each particular social influence world contains some signal relating to the underlying appeal of the songs and some noise that results from the interdependence of decision making. When the social influence is stronger, as in experiment 2, that noise appears to be greater. We speculate that if one could generate thousands of social influence worlds and average their outcomes that this average success would be strongly related to the underlying appeal of the products. However, in practice we cannot observe these parallel worlds, and the ratio of signal to noise in the outcomes in real cultural market is an open question.

Overall, the results from experiments 1 and 2 provided strong support for the argument that social influence at the individual level is simultaneously responsible for increased inequality and unpredictability in collective outcomes—in this case, the distribution of market share. Although simple to state, this finding nevertheless exhibits a curious paradox: on the one hand, by revealing the existing popularity of songs to individuals, the market provides them with real, and often useful, information; but on the other hand, if they actually use this information, the market inevitably aggregates less useful information. This result, which is analogous to “information cascades” in economics (Banerjee, 1992; Bikhchandani et al., 1992), suggests, in turn, that social institutions that make us aware of the behavior of others—the New York Times bestseller list, the Billboard album charts, and lists of top-grossing movies—do provide a useful service to individuals, but only at the cost of increasing the overall inequality and unpredictability of the markets themselves.

6. A new pool of participants

There are risks, however, in drawing conclusions from these studies that were based mostly on the behavior of American teenagers, a population that may be unusual in many ways. For example, there is some evidence that younger people may be more susceptible to social influence (Park & Lessig, 1977; Pasupathi, 1999) and that conformity levels vary by country (Bond & Smith, 1996). For this reason we now investigate how the individual and collective level results that we observed in experiment 2 would change if we used a different group of participants. Further, by observing the same process with two distinct pools of participants we were able to make additional comparisons that are not possible with just one pool of participants.

To recruit this different set of participants we sent 13,546 emails to subjects from a previous, unrelated web-based study (Dodds, Muhamad, & Watts, 2003). Given the nature of the Dodds et al. (2003) study, we expected these participants to be quite different than the participants recruited from bolt.com. From March 14, 2005 to April 7, 2005 (24 days), these emails, plus the web-postings they generated, yielded a pool of 2,930 participants that were assigned into two social influence worlds and one independent world, corresponding to a similar number of participants per world as in experiments 1 and 2.¹⁵

Most important for the goals here, these participants differed from those in experiment 2 in a number of ways, as we had expected. First, the two groups were different demographically with participants in experiment 3 being older, more male, and more international (Table 1). Second, the participants differed in their behavior—in particular, participants in experiment 3 listened to twice as many songs, on average, as those in experiment 2, a difference we had not anticipated (Table 2). Finally, participants in experiment 3 had different musical preferences than those in experiment 2. Figure 7A compares the appeal measures from experiment 2 and 3 and shows that the different pools of participants had slightly different musical preferences (r = 0.37). Further, participants in experiment 3 liked the songs less on average than the participants in experiment 2 as can be seen in Figure 7B, which compares the probability that a listen for a specific song will lead to a download—what Aizen et al. (2004) call the “batting average.” Overall, 46 of the 48 songs had lower batting averages in experiment 3;¹⁶ a finding that is consistent with the lower song ratings that were reported in experiment 3 (mean of 2.6 stars) compared to experiment 2 (mean of 3.3 stars) (Table 2).¹⁷ Thus, we conclude that, as expected, the participants recruited for experiment 3 differed along at least three important dimensions: demographics, behavior, and musical preferences.

Table 2.

Descriptive statistics of participant behavior in experiment 2 and 3.

	Experiment 2			Experiment 3
	Social Inf. 8 worlds (n=5,746)	Indep. 1 world (n=1,446)	Total 9 worlds (n=7,192)	Social Inf. 2 worlds (n=1,471)	Indep. 1 world (n=1,459)	Total 3 worlds (n=2,930)
Listens	20,217	5,643	25,860	10,591	11,844	22,435
Listens per participant	3.5	3.9	3.6	7.2	8.1	7.7
Downloads	8,106	2,192	10,298	2,040	1,691	3,731
Downloads per participant	1.4	1.5	1.4	1.4	1.2	1.3
Pr[Download | Listen]	0.401	0.377	0.398	0.193	0.143	0.166
Mean rating (# of stars)	3.31	3.22	3.29	2.70	2.55	2.62

Figure 7.

Comparing the preferences of the participants in experiment 2 and 3.

In spite of these differences, participants from experiment 3 were influenced by the behavior of others in a similar way to participants in experiment 2. Figure 8 plots the probability that a participant will listen to a song of a given rank (independent of which song occupied that rank at that time). Participants in experiment 3 had higher listening rates overall, and thus all the effects observed in experiment 2 are more pronounced in experiment 3. In qualitative terms, however, the impact of social influence, as measured by the difference between independent (open circles) and social influence (closed circles) conditions, was similar in both experiments.

Figure 8.

Probability of listening to a song of a given market rank. The patterns in experiment 2 and 3 were qualitatively similar.

Figure 9 shows that the results at the collective-level were also qualitatively similar. For example, in both experiment 2 (9A) and experiment 3 (9B) the inequality of success in the social influence worlds was greater than the inequality in the independent world (p < 0.01). Further, a comparison across experiments shows the amount of inequality in the social influence worlds was similar to the amount of the inequality in the social influence worlds in experiment 2. The independent worlds, however, appear to have different levels of inequality: G=0.19 in experiment 2 and G=0.27 in experiment 3. We suspect that this increased inequality in the independent condition can be attributed to the greater variation in the probability of download given listen in experiment 3 than in experiment 2 (Figure 6B). Figures 9C and 9D show that the full dynamic trajectories—not just the steady states—were similar across experiments.

Figure 9.

Inequality of success in experiments 2 and 3. The inequality of success in experiment 3 was similar to the inequality of success in experiment 2.

Considering now the unpredictability of success, Figure 10 shows that in both experiment 2 (10A) and experiment 3 (10B), the unpredictability of success was greater in social influence condition than the independent condition (p < 0.01). A comparison across experiments, however, shows that the unpredictability in the social influence condition seems to have decreased from experiment 2 to 3. We suspect that this difference could be because participants in the social influence condition in experiment 3 listened to so many more songs than those in experiment 2: 7.2 compared to 3.5. This extensive listening behavior ensured that all songs had a reasonable number of listens and therefore prevented the high appeal songs from escaping notice. Finally, Figures 10C and 10D show that—as with the inequality measure above—the dynamics of the unpredictability measure were qualitatively similar in both experiments.

Figure 10.

Unpredictability of success in experiments 2 and 3. The unpredictability of success in experiment 3 was similar to the unpredictability of success in experiment 2.

Collectively, these three experiments illustrate an important point made by the sociologist Stanley Lieberson (1997): when considering interventions to a social system it is important to distinguish between changes that will affect the distribution of outcomes while preserving relative positions, and changes that, by contrast, will affect relative positions while keeping the distribution of outcomes unchanged. For example, although the aggregate features of experiments 2 and 3 were similar, the results for specific songs were different (as might be expected given the differences in preferences). Consider “Life's Mystery,” an acoustic ballad by the singer Dante; this song did poorly with the mostly teen-age participants in experiment 2—it was in the bottom half of popularity in 6 of 8 influence worlds. However, in experiment 3, with an older, more international population of participants, “Life's Mystery” fared much better, coming in 6th and 5th in the two influence worlds. In general, if we calculate K̅_i, the average market rank of song i in all of the social influence worlds in an experiment, the correlation between this measure in experiment 2 and experiment 3 was 0.52. Thus, similarity of results at the aggregate level between experiment 2 and 3 was coupled with some change in the ordering of the songs.

A comparison between experiments 1 and 2 reveals a different pattern. In these experiments there was little change at the song level; the correlation between average market rank in the social influence worlds of experiment 1 and 2 was 0.93.¹⁸ In other words, songs that did well in experiment 1 tended to do well in experiment 2, as one might suspect given that participants in both experiments were similar. However, changing the song menu lead to large changes at the aggregate level with the Gini coefficient increasing from G ~ 0.34 to G ~ 0.50. Together therefore, these experiments show that one of our interventions, increasing the salience of the popularity of the songs (experiments 1 and 2), lead to stability in the ordering of the songs coupled with changes in the aggregate features of the market. Our other intervention, changing the pool of participants (experiments 2 and 3), lead to stability in the aggregate features of the market coupled with changes in the ordering of the songs.

7. Predicting success

The relationship between appeal and success shown in Figure 6 suggests that one might reasonably think of outcomes that are observed in the social world as arising out of a combination of both individual preferences and aggregation dynamics. Predicting outcomes, therefore, requires one to understand not only the distribution of individual preferences in the target population, but also to intuit how all those preferences will aggregate when everyone is also paying attention to what other people are doing. We have already argued that it is this second component to success that is inherently unpredictable; but our comparison of experiments 2 and 3 raises a related question—namely how much can one learn from the results of one experiment that could be applied to another? This question has a practical analog in business. For example, it is often the case that an executive contemplating the launch of an object X (say a book or a TV show) in market B can observe the success of the same object in some other market A in which it was launched previously. Given that the two markets are presumably characterized by different distributions of preferences, the question, then, is how much weight should the executive place on X’s success in market A relative to her own pre-market audience testing in market B?

Our experiment was not designed to answer this particular question, and so it can offer no definitive answer. Nevertheless, we can examine our data in order to gain a qualitative sense of the relative predictive value of these two kinds of information in our experiment. We begin by calculating the prediction error,

$$\epsilon =\sum _{s=1}^{48}|m_{sw}-{\mathrm{m̂}}_{sw}|$$

defined as the total (over all songs) difference between the predicted and actual market shares, where m_sw is the market share of song s in world w, and m̂_sw is the predicted market share of song s in world w. In this case, the predicted market share will be defined to be the market share that has been observed; in the hypothetical example this could be the market share in either market A or in pre-market testing in market B. Unfortunately, ε is hard to interpret directly, but it ranges from a minimum of 0, if the prediction is perfectly accurate, to a maximum of 2. To improve understanding, therefore, we can give these results a proportional reduction in error interpretation by comparing the prediction using our method and a given piece of information to the prediction that would be made using no information. In this case, the prediction with no information would be that each song would earn the average market share,

$${\epsilon }_{\mathit{\text{naive}}}=\sum _{s=1}^{48}|m_{sw}-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$48$}\right.|$$

By comparing the errors with different prediction methods we can give our predictor a proportional reduction in error interpretation (Costner, 1965),

$$PRE=\frac{{\epsilon }_{\mathit{\text{naive}}}-\epsilon }{{\epsilon }_{\mathit{\text{naive}}}}$$

This measure varies from −∞ to 1, where a negative value means that naive predictions are better than our predictions, a value of 0 means that both predictions have equal error, and a value of 1 means that our predictions have no error. We can illustrate the measure with our hypothetical example of a business executive who wishes to predict market share in a social influence world in experiment 2.¹⁹ Using the naive prediction, her error would be 0.77;²⁰ however, if she also has access to extensive pre-market testing (analogous to knowing the distribution of popularity in the experiment 2 independent condition) then her expected error is 0.65. Her expected error, in other words, has been reduced by about $\raisebox{1ex}{$(0.77-0.65)$}\!\left/ \!\raisebox{-1ex}{$0.77$}\right.=0.16$ or 16%, leaving the remaining 84% of the error unexplained. Even knowing the full distribution of popularity in the independent condition is therefore of limited value to someone attempting to predict outcomes in the presence of social influence—a result consistent with our general claim that some portion of the unpredictability of market success cannot be eliminated even with perfect information about the pre-existing preferences of individuals.

Having absorbed the importance of social influence on individual decision making, one might suspect that evidence of “real world” success (or failure) in a different market would be more informative than an independent sampling of the audience in question. Yet in fact, the opposite is true: the proportional reduction in error from using outcomes from a social influence world in experiment 3 to predict success in a social influence world from experiment 2 is a paltry 2%. Performing the same exercise with experiment 3, moreover, reveals the same result, but with an even more striking contrast. Whereas knowing the distribution of popularity in the independent condition of experiment 3 yields a reduction in error of 38% when predicting outcomes in a social influence world in experiment 3, using a social influence world from experiment 2 to predict success in a social influence world in experiment 3 yields an increase in error of 3% relative to the naïve prediction.²¹ In other words, in this situation our hypothetical executive would do just as well simply guessing.

That past performance does not guarantee future returns may not seem like an overly surprising result, and no doubt many examples can be found where a book, movie, or song that has been successful in one market had been turned down by executives on the basis that “it wouldn’t work over here.” Indeed it was presumably on this basis that mega-hit American Idol was rejected by TV executives in the US in spite of the popularity of its progenitor, Pop Idol, in the UK. On the other hand, following on the American success of the British game show Who Wants to be a Millionaire?, NBC eagerly aired The Weakest Link, another game show popular in the UK; it went on to flop (Carter, 2006).

In light of the many complexities involved, we wish to avoid placing too much emphasis on these results, which are purely suggestive.²² Nevertheless, the result that one could learn virtually nothing from one market outcome that could be applied in another was, we feel, a surprising result. Indeed, as indicated in Figure 11 (which summarizes all the comparisons above) even the independent conditions from the “wrong” experiment were more informative than the social influence outcomes (compare the diagonal arrows with the vertical arrows on the right hand side). The results from Figure 6, however, provide at least some intuition for these findings. If we consider the outcomes of a social influence world to be noisily related to appeal, then it stands to reason that one noisy outcome can’t predict another noisy outcome, especially when the underlying signal itself may be changing. We hope that future modeling or experiments will address these issues in more detail.

Figure 11.

Proportional reduction in error in predicting success. Having information about the success in a social influence world in one experiment is essentially useless in predicting success in a social influence world in the other experiment

8. Discussion and next steps

The experiments presented here provide additional support for the results in Salganik, Dodds, and Watts (2006) suggesting that both inequality and unpredictability in cultural markets inevitably arise from an individual-level process of social influence. Using a pool of participants with different demographics, behavior, and preferences, the results of experiment 3 were qualitatively similar to those from experiment 2. However, a comparison of the two experiments also reveals that our qualitative understanding of the process is of little use in explaining the differences between experiments 2 and 3. There are several things that are different about the participants and we are not sure which of these lead to the similarities and differences in collective-level results that we observed. Emphasizing the preliminary nature of our conclusions, therefore, we end by outlining some possible next steps.

One natural next step would be to formulate a mathematical or agent-based model of this process. Agent-based models are wonderful tools for studying collective behavior, but they are often limited by a lack of empirical foundations at either the individual or collective level (Epstein & Axtell, 1996; Goldstone & Janssen, 2005; Macy & Willer, 2002).²³ We believe that these experimental results are ideal for guiding agent-based modeling because they provide both detailed individual-level data that can be used to choose between alternative micro-foundation and detailed collective-level patterns that can then be used to validate the model. Once researchers are able to produce a simulation model that can reproduce the observed collective behavior (both dynamics and steady-state), they can then explore how different parameters affect the observed collective dynamics. An example of this approach of linking group experiments to agent-based modeling is the work done on foraging patterns by Goldstone and colleagues (Goldstone & Ashpole, 2004; Goldstone et al., 2005; Roberts & Goldstone, 2006). With a fully calibrated model one can ask many questions. How would the results have been different if the social influence was increased? What if the songs were more similar (or different) in terms of their appeal? What if there were 48,000 songs instead of 48? These are all important questions because the experiments presented here represent only three data points in the set of all possible experiments with this design.²⁴ Careful simulation modeling could suggest interesting predictions about other portions of this parameter space that can be tested with further experiments, and some work in this direction has already begun (Borghesi & Bouchaud, 2007).²⁵

In addition to other experiments using similar designs, we suspect that many future experiments in large-group collective dynamics will also take place on the web. These experiments will pose specific challenges above and beyond the challenges of web-based experiments in general (Kraut et al., 2004; Nosek, Banaji, & Greenwald, 2002a; Reips, 2002; Skitka & Sargis, 2006). The experiments performed here allowed for a very limited form of interaction in the sense that participants made one-time decisions based on the prior decisions of others. By contrast, many social situations exhibit more complex interaction where individuals are constantly adjusting their behavior based on the behavior of others. In an experimental situation when this decision-making is happening in continuous time, all experimental subjects must be participating at exactly the same time. This demand for simultaneity, which was not present in the experiments presented here, introduces tremendous logistical difficulties. Some of these difficulties can be overcome by placing all subjects physically in the same lab at the same time —for example, in the conformity experiments of Latane and L’Herrou (1996), and the graph coloring experiments of Kearns et al. (2006)—but then the difficulty of scaling beyond small groups again becomes an obstacle. Lifting the scale constraints, therefore, requires a true web-based “platform” for conducting experiments, but it is not yet clear how such a platform would handle complex coordination of subjects, participant dropout, and other issues that are relatively easily resolved in the close physical proximity of a physical lab.²⁶

Finally, another limitation to scaling to truly large groups is that most experiments currently run by social scientists are not intrinsically enjoyable for participants; thus subjects are generally paid in exchange for their participation. Although paying people is certainly reasonable, at just $8 per subject, the three experiments presented here—which collectively involved 17,271 subjects—would have required about $138,000 just for subject compensation. To run large experiments on the web, therefore, researchers will need to develop new models of compensation, such as micro-payments on “crowd-sourcing” platforms like Amazon’s Mechanical Turk, or, as we have done here, by designing experiments that are intrinsically rewarding for participants (Dodds et al., 2003; Nosek, Banaji, & Greenwald, 2002b; Srivastava, John, Gosling, & Potter, 2003; von Ahn & Dabbish, 2004). It remains to be seen what types of incentive schemes will emerge for web-based experiments, but social scientists have proven adept at experimental design in the past, and we suspect that there will be a number of clever experiments deployed over the web in the next few years. Regardless, the web offers exciting possibilities for the experimental study of collective social dynamics, and we hope these preliminary results will encourage other researchers to explore the web not only as an object of sociological interest in itself, but a tool for doing social science on a scale only recently imaginable (Watts, 2007).