Hidden Processes in Structural Representations: A reply to Abbott, Austerweil, and Griffiths

Abstract

In recent work exploring the semantic fluency task (Hills, Jones, & Todd, 2012) we found evidence indicative of optimal foraging policies in memory search that mirror search in physical environments. We determined that a two-stage cue-switching model applied to a memory representation from a semantic space model best explained the human data. Abbott, Austerweil, and Griffiths (in press) demonstrate how these patterns could also emerge from a random walk applied to a network representation of memory based on human free association norms. However, a major representational issue limits any conclusions that can be drawn about the process model comparison: Our process model operated on a memory space constructed from a learning model, while their model used human behavioral data from a task that is quite similar to the behavior they attempt to explain. Predicting semantic fluency (e.g., how likely it is to say cat after dog in a sequence of animals) from free association (how likely it is to say cat when given dog as a cue) should be possible with a relatively simple retrieval mechanism. The two tasks both tap memory, but they also share a common process of retrieval. Assuming that semantic memory is a network from free association behavior embeds variance due to the shared retrieval process directly into the representation. A simple process mechanism is then sufficient to simulate semantic fluency because much of the requisite process complexity may already be hidden in the representation.

The semantic fluency task (SFT; e.g., “name all the animals you can in a minute”) is widely used¹ to study memory retrieval in both experimental and clinical settings. A key finding in SFT is that participants tend to produce temporal bursts of semantically related items with longer lags between bursts. This “patchy” response pattern has led to the proposal that memory retrieval in the task is the product of two distinct processes: a local search process that generates a series of related items based on inter-item similarity, and a global process that moves between local regions of the search space when these regions become depleted (e.g., Troyer, Moscovitch, & Winocur, 1997; Gronlund & Shiffrin, 1986).

In previous work (Hills, Jones, & Todd, 2012), we compared the response patterns in SFT to patterns seen when animals are foraging for food, and found evidence that humans searching memory in SFT produced statistical signatures that are characteristic of optimal foraging policies observed in animals searching for resources in physical space (Charnov, 1976). This correspondence suggests that the mechanisms for searching memory may have been exapted from mechanisms that evolved to forage for resources in the physical world (Hills, 2006; Hills et al., 2015). However, there are alternate explanations that may also account for the behavioral patterns: It may simply be the case that memory is organized in such a patchy fashion that a model randomly producing items in relation to their similarity to previously produced items would naturally generate data that appear indicative of optimal foraging.

We tested a variety of models of search, based on classic cue-combination memory search models (Anderson, 1990; Raaijmakers & Shiffrin, 1981) applied to a spatial representation of semantic memory constructed by a corpus-based distributional mechanism (BEAGLE; Jones & Mewhort, 2007). The model that best explained the human data was one which was able to switch between two cues: Local similarity was used to generate items until no other item was close enough to pass a threshold, and then the model switched to a global frequency cue to select the next item (and search reverted again to local similarity).

Abbott, Austerweil, and Griffiths (in press; hereafter AAG) point out historical issues of model “mimicry” that may well be at play in our analysis. Our findings may be dependent on the memory representation we used.² Furthermore, our rejection of a one-stage random selection model, in favor of a two-stage local vs. global switching model, is potentially at odds with the success of random walk models in other areas of psychology (e.g., Griffiths, Steyvers, & Firl, 2007). To illustrate the issue of model mimicry, AAG first apply a random walk model to a network representation constructed from our BEAGLE space, and replicate our finding that such a model is insufficient to simulate the human data. They then apply a random walk to a network representation constructed from free association norms (Nelson, McEvoy, & Schreiber, 2004) and find that this model reproduces the behavioral markers indicative of optimal foraging that our local/global cue-switching model is able to. They propose that a random walk over a free-association network is an alternative account of SFT that is equally plausible to our cue-switching model over a semantic space.

AAG analyzed the representations used by each model (BEAGLE space vs. free association network) and determined that the behavioral bursts in SFT were better represented as clusters in the association network than they were in the BEAGLE space, and this embedded structure is what allowed their random walk model to mimic the optimal foraging predictions of our cue-switching model. They conclude, “a random walk on a semantic network produces optimal foraging behavior, but a random walk on a corresponding spatial representation does not. Consequently, it seems that there is something special about the semantic network representation that allows the simple random walk to appear similar to optimal foraging.”

We certainly concur with AAG that this is a research area ripe for model comparison, and that constructing competing models of SFT is important to help design new experiments that constrain theoretical accounts and home in on the true generating process. However, the competing accounts being compared must be equivalent or no firm conclusions can be drawn and we risk rejecting what may well be an excellent model of the processes and representations that humans actually use. For example, the casual reader may conclude from the AAG article that a network representation of memory is more like what humans have in their heads than a spatial model, and Occam’s razor then favors the more parsimonious random walk as the process that humans use when retrieving items in SFT over our local/global cue-switching model. The free-association network is a model of memory structure that better represents the behavioral clusters seen in SFT, and this allows a simple random walk to successfully produce patterns consistent with optimal foraging.

But there is a deep and broadly important issue concerning the structural representations of memory, and the “something special” that AAG suggest about the semantic network used in their model compared to the spatial representation used in ours. Simply put, the spatial representation was based on a statistical model of learning, whereas the network representation was quite literally built from human behavioral data on a task similar to SFT. The process used in SFT is already partially embedded in AAG’s network representation.

Hidden Process Complexity in the Structural Representation

A model of a cognitive phenomenon requires an account of both structure and process, and how the two interact (Estes, 1975). Too often in cognitive modeling, the structural representation is ignored in favor of the process account (Johns & Jones, 2010). AAG are explicit in their model that memory is an associative network over which a random walk operates. Similarly, we were explicit that memory in our model is a spatial representation over which a local/global search process operates. But these two proposals differ in a crucial way.

BEAGLE (Jones, Kintsch, & Mewhort, 2006; Jones & Mewhort, 2007) is a mechanistic model of how humans construct a semantic space from statistical redundancies in the environment (trained on a natural language corpus). In contrast, free association is a behavioral task. The free association norms (Nelson et al., 2004), used by AAG to construct their network, are aggregated human responses to cue words—behavioral data. BEAGLE explains, for instance, how the pair dog-cat become similar to one another in memory by applying a learning mechanism to environmental regularities that humans learn from. Indeed, in early language development the structure of actual language is a better predictor of word learning than are free association norms (Hills, Maouene, Riordan, & Smith, 2010; Hills, 2012).

Free association norms are simply a tabulation of how often humans give, for instance, cat as a first response to dog as a cue. Free association is not a pure readout of the structure of semantic memory—it also incorporates the process of memory retrieval. A concern then is that AAG’s “representation” of memory is based on human responses from an association task that is already very similar to the SFT that the model is meant to explain. Hence, much of the requisite variance needed to account for the process of SFT may already be embedded in the representation. As we describe in the next section, the problem here—proposing that memory is represented as what people do in a task very similar to SFT, and then concluding that a simple process model is sufficient to simulate the SFT behavioral data—is an instance of what we refer to as a Turk problem in cognitive modeling. Turk problems are so-named in reference to the 18th century chess-playing automaton, and we use it to express the “man in the machine” issue of using human behavioral data as a cognitive model’s mental representation. Rather than explaining anything, this approach primarily predicts one kind of behavior (here SFT) from another very similar behavior (free association).

Free association is a dependent variable—it is something that we need to explain, not a pure measure of how memory is organized. Nelson and McEvoy have noted this in virtually all of their publications on free association, but the practice of using their association norms as a proxy for human memory is still widespread. For example, Nelson et al., (2004) originally emphasized the potential of using free association norms “for evaluating models of semantic representation” (p. 403), and noted that free association is a good predictor of many memory tasks because it uses some of the same retrieval processes used in those tasks, as well as tapping some commonalities of semantic memory. Although free association was intended as a dependent variable, Nelson, McEvoy, and Dennis (2000) note that its primary use is as a yardstick for stimulus control—but very little work has been done to investigate the yardstick itself: “Unlike printed frequency norms in which words are counted across samples of text, words are produced in free association, and we know little about the representations and processes involved” (p. 888).

The tasks of free association and SFT are extremely similar, not only because they both tap semantic memory representations, but also because they draw on similar retrieval processes. The similarity is clear in AAG’s analysis showing that the clusters in the free association network are already very similar to the SFT response clusters. SFT can be thought of as a continuous version of the free association task constrained to a given category (animals). In fact, Bousfield and Sedgewick (1944) originally referred to the task as “sequences of restricted associative responses.” In introducing the task, they note that it “is essentially an extension of the classic word association” task (p. 149).

Consider the argument ad absurdum: An even simpler zero-parameter model that perfectly accounts for the SFT data can be built based on the assumption that human memory is represented as an associative chain using data from SFT. This is obviously absurd because it suggests that no process is required, and just simulates SFT data from SFT data. While it is a very parsimonious and predictive model, it has no explanatory power. Predicting behavior from similar behavior without generating new understanding is the problem underlying the idea that SFT can be simulated from free association data.

Hence, we do not believe that the critical difference between our model and the AAG model is a space versus a network or a random walk versus a cue-switching model. From a graph theoretic perspective, spaces are networks—fundamentally, both are matrices. Moreover, as noted above, both models use random walks over local proximity-based representations. The real issue is, in AAG’s words: “…a random walk on a semantic network derived from free association produces phenomena suggestive of optimal foraging, while a random walk on a spatial representation generated by BEAGLE does not. This raises the natural question: Why? What is the critical difference between these two representations?” Our response is simple: much of the variance required to simulate the SFT process is already partially embedded in the free association representation.

BEAGLE may very well be wrong as a model of human semantic representation. But a much more informative model comparison would be to pit BEAGLE (a spatial model) against a network created with a Topic model (Griffiths, Steyvers, & Tenenbaum, 2007)—Topic models have been used to generate networks, and are a natural contender. Then, both learning models could be applied to the same experience (a text corpus), and the resulting network and space could be fully crossed with a random walk or cue-switching model.

Notably, the problem of inferring process without attending to the influence of the search environment has also led to controversy in the ecological search literature which inspired this cognitive work. Lévy flights have been proposed to parsimoniously model animal foraging because, not surprisingly, sampling from a power-law distribution of path lengths produces the power-law distribution of path lengths that animals often produce when foraging for food. However, closer inspection of animal behavior and the resource environments in which they forage has found that animals change their behavior to increase time in resource-rich environments while minimizing time in resource-poor environments. This behavior (often called area-restricted search) also produces power-law distributed path lengths (Hills, Kalff, & Wiener, 2013; Plank & James, 2008; Benhamou, 2007) and is more optimal than Lévy flights (Plank & James, 2008; Ferreira, Raposo, Viswanathan, & Luz, 2012). Hence, assumptions that overly simplify the search environment are likely to get the search process wrong.

The Representational Turk Problem in Cognitive Modeling

As indicated above, the practice of using free association norms as a proxy for human semantic memory has a considerable history in cognitive modeling (e.g., De Deyne, Navarro, & Storms, 2013; Steyvers, Shiffrin, & Nelson, 2004). But it is important to remember that any representation based on behavioral data also contains variance from the process used to produce the response. Oddly, the opposite problem is seen in the semantic representation literature: Complex learning models are used to generate estimates of memory structure, but then a remarkably simple metric (e.g., vector cosine) is used to predict human data (see Jones, Willits, & Dennis, 2015 for a review). But a cosine is certainly not a process model. Hence, the semantic representation literature needs to likewise consider process accounts (retrieval, similarity judgments, etc.) when fitting representational models to human data.

Examples of embedding human behavior in a model to then predict similar human behavior are easy to find across the field. For example, Glenberg and Robertson (2000) demonstrated the superiority of their indexical hypothesis over Latent Semantic Analysis (LSA; Landauer & Dumais, 1997) at explaining human comprehension of sentence affordances. They found that LSA could not distinguish between afforded sentences (using a newspaper to protect one’s face from the wind) and non-afforded sentences (using a matchbook to protect one’s face from the wind), but this distinction was naturally predicted by their indexical hypothesis. Humans rated the afforded sentences as easier to “envision” than the non-afforded sentences. However, the affordance predictions of Glenberg and Robertson’s account were generated from sensibility ratings (virtual nonsense to completely sensible) made by the same subjects who produced the envisioning ratings (impossible to imagine to easy to imagine) that were used as the dependent variable. Not only were the two ratings on judgments that are almost identical, they were from the same participants. Hence, it is both impossible for a learning model like LSA to outperform humans, and is questionable what explanatory power comes from such an exercise.³

Certainly, the use of norms as a proxy for human representation has a very fruitful history in cognitive modeling (e.g., McRae, de Sa, & Seidenberg, 1997; Smith, Shoben, & Rips, 1974). But the representational Turk problem is becoming more common in cognitive modeling as we strive for parsimony under threat of Occam’s razor. We have formal methods for penalizing models based on parametric complexity (Lewandowsky & Farrell, 2010). However, we only penalize the process account; the field does not yet have a good idea how to “tax” a model based on its representational assumptions. The representation is becoming the Cayman Islands of modeling, where complexity may be hidden from such taxation to support a simpler process mechanism, often unintentionally. Hummel and Holyoak (2003) have gone so far as to suggest that representational complexity may be the single most serious problem facing cognitive modeling as a scientific enterprise. Particularly as we see more large-scale models making use of naturalistic ‘big data’ in their representations (Griffiths, 2015; Jones, 2015), it is important to be vigilant about process variance hiding in the representation.