Single-Case Cognitive Neuropsychology in the Age of Big Data

Abstract

Historically, single-case studies of individuals with brain damage have contributed substantially to our understanding of cognitive processes and their neural substrates. However, the role of single case cognitive neuropsychology has diminished with the proliferation of techniques that measure neural activity in humans. Instead, large-scale informatics approaches in which data is gathered from hundreds of neuroimaging studies have become popular. Furthermore, it has been claimed that utilizing these informatics approaches can address problems found in single imaging studies. We first discuss reasons for why cognitive neuropsychology is thought to be in decline. Next, we note how these informatics approaches, while having benefits, are not particularly suited for understanding functional architectures. We propose that the single-case cognitive neuropsychological approach, focused on developing models of cognitive processing, addresses several of the weaknesses inherent in informatics approaches. Furthermore, we discuss how using neural data from individuals with brain damage provides data that can inform both cognitive and neural models of cognitive processing.

The primary goal of cognitive neuropsychology is to use evidence from brain-damaged individuals to draw inferences about the organization of the normal cognitive system. In many ways, evidence from brain-damaged individuals allows us to make significant advances in our understanding of cognition. One clear benefit of cognitive neuropsychological research, especially single case studies, is the serendipitous nature of these explorations. The experimenter has no control over lesion location, and has to figure out how to characterize the individual’s cognitive impairment. Sometimes, their behavior fits well with existing cognitive models. But at other times, the location of the stroke results in patterns of behavior that cannot be explained based on current models. When presented with these unexpected behaviors, the experimenter is led down a path of experiments that can end in the development of new cognitive models. These cases can often result in major breakthroughs in our understanding of cognition. A number of “textbook” cases have had substantial impact on our understanding of cognition, including memory (Scoville & Milner, 1957), vision and action (Goodale & Milner, 1992), language (Broca, 1861; Wernicke, 1874), reading (Marshall & Newcombe, 1966), visuospatial attention (Bisiach & Luzzatti, 1978; Caramazza & Hillis, 1990a), spatial representations (McCloskey et al., 1995), and others (Caramazza, 1986; Coltheart, 2001; McCloskey & Chaisilprungraung, 2017).

Cognitive neuropsychology – A field in decline?

Although cognitive neuropsychology has provided ample evidence to develop our understanding of cognitive architectures, there is a sense that it is in decline as a method of scientific inquiry. Some journals that previously published single case studies now have explicit policies against it¹ or will only consider extremely rare cases². Others have noted that cognitive neuropsychology is considered by some as a “relic of a past era” (Rorden & Karnath, 2004), and have debated whether the cognitive neuropsychology research paradigm as a “dodo or phoenix” (Shallice, 2014). Along with Shallice, we do not believe that cognitive neuropsychology is a “dodo” that has outlived its usefulness as a method to understand cognition. Though there have been some critiques of the approach (e.g. Patterson & Plaut, 2009), this decline cannot simply be attributed to theoretical papers pointing out the flaws in the method. Evidence from cognitive neuropsychological studies are still widely cited and taught regularly as providing evidence for cognitive theories, and the assumptions that underlie the approach are still valid (see Coltheart, this issue).

Although we argue that cognitive neuropsychology has not declined in its usefulness, there has been a clear decline in its usage since its peak (Chatterjee, 2005; Fellows et al., 2005). We contend that this decline is due to difficulties specific to patient work and sociological factors. First, there are a number of barriers to cognitive neuropsychological research that are not found in other methods. Some of these barriers have to do with the special nature of the population being tested. Finding brain-damaged individuals who are willing to engage in substantial amounts of research is difficult even with access to a clinical population (Fellows, Stark, Berg, & Chatterjee, 2008). For those interested in using evidence from brain-damaged individuals to understand neural correlates of functions (e.g. voxel-lesion symptom-mapping), it is difficult to recruit a large enough sample for sufficiently powered studies (Kimberg, Coslett, & Schwartz, 2007). Other barriers concern the enormous effort typically required for collecting data within this approach. In single case cognitive neuropsychology, it is often necessary to generate and run a number of experiments, many specifically for testing this single individual, to isolate a deficit to a specific function. Administering all of these experiments frequently takes months or even years. Furthermore, if that single case has another stroke or health event before the study is complete, which is common when working with an older population with pre-existing cardiovascular problems, a significant outlay of effort and resources can be for naught. Even if the participant is able to complete every experiment, the substantial outlay of effort typically results in only one publication. Given pressures to publish and a competitive grant climate, the incentives are skewed towards maximizing the number of publications. With this in mind, many will opt towards methods that involve less outlay and risk.

Second, there are strong incentives for using the newest technique, as there are clear rewards to be the first individual to report a specific finding with a new technique (i.e. increased likelihood of getting a grant, publication in higher impact journals). Given that nearly all methods used in the cognitive neurosciences (PET, TMS, tDCS, various types of functional and structural neuroimaging methods and techniques) are more recent than cognitive neuropsychology, the field suffers from being perceived as a “relic”. Third, given that the major technological advances in cognitive neuroscience have been in neuroimaging, there has been a large shift in the field to focusing on neural activity over developing cognitive models. For example, Shallice (2009) identified two strands in cognitive neuroscientific research; a biomedical strand with its historical origins in neurophysiology with a focus on neural substrates, and a more cognitive strand focused on theories of information processing. Early functional neuroimaging work found that certain cognitive processes were associated with activity in specific brain regions – a sensible initial endeavor for studying structure-function relationships. These initial studies were highly cited and published in high impact journals. Given that the first study to show that process X was associated with brain region Y were rewarded, this created skewed incentives to grab the remaining low-hanging fruit – finding the neural correlates of other processes, tasks, constructs, and now famous pop songs (“Gangnam Style”, Chen et al., 2017), Harry Potter (Hsu, Jacobs, Altmann, & Conrad, 2015), and political attitudes (Kaplan, Freedman, & Iacoboni, 2007). Again, this created an initial framework in which studies that used neural data to adjudicate between cognitive theories were not as incentivized. All of these factors have contributed to a decline in focus and resources on cognitive neuropsychological (and also cognitive psychological) research. These disincentives disproportionately affect younger researchers. Established cognitive neuropsychologists have already established a name for themselves, and are protected (at some level) by their reputation and tenure. However, the ability of younger researchers to obtain a post-doctoral position, faculty position, and tenure is primarily dictated by publication count and manuscripts in high-impact journals. Given this, young researchers will be less likely to use cognitive neuropsychological methods, leading to fewer publications and a field that is in decline.

Big data – More peril, less promise?

Paired with these disincentives is a growing feeling in cognitive science that we need to rely more on “big data” from hundreds of individuals, which would entail a move away from the single case approach common in cognitive neuropsychology (see Coltheart, 2001 for a discussion of replicability in cognitive neuropsychology). Researchers are rightly concerned about how small sample sizes common in behavioral and neuroimaging increase the likelihood of false positive results and decrease the likelihood of replication (Button et al., 2013; Ioannidis, 2005; Szucs & Ioannidis, 2017; Yarkoni, 2009). One option for addressing this concern would be to increase the sample size in each individual study. An alternative, which has recently been favored, is to synthesize results via meta-analysis. Given that neuroimaging has become the dominant tool in the cognitive sciences, there has been an explosion in the number of published fMRI papers (with one estimate at 40,000; Eklund, Nichols, & Knutsson, 2016), and terabytes of data generated by these papers. The ability to share neuroimaging data, along with automated methods to extract data from the literature and increases in computing power, has led many to suggest that “big data” approaches will be transformative for understanding cognition. Specifically, it has been argued that these informatics-driven approaches address a myriad of weaknesses inherent in single neuroimaging studies – issues of statistical power, reverse inference and experimental design - leading to a better understanding of structure-function mapping. Once these structure-function mappings are well understood, then neural data could be used to develop cognitive theories (Henson, 2005; Mather, Cacioppo, & Kanwisher, 2013). However, as we will argue below, the promises of these big data approaches may be overstated, and scientific advances with these techniques will require the kind of careful inferential work cognitive neuropsychologists use when developing cognitive theory from the pattern of performance of a single case study.

Consider, for example, the Neurosynth system (www.neurosynth.org) developed by Yarkoni and colleagues (2011), which has been used to generate hundreds of fMRI meta-analyses. Yarkoni and colleagues (2011) used a text mining approach to extract reported significant activations and important terms from over 11,000 studies. The Neurosynth system allows one to examine what brain regions are consistently or preferentially active for a specific keyword, or to examine what keywords are most associated with activation of a specific voxel. In addition to addressing issues of statistical power, large databases such as this could be used to deal with the issue of reverse inference, using thousands of studies to see if a specific brain region is associated with only one, or several cognitive processes. At a coarse level, this approach provides interesting information regarding the relationship between brain activity and general cognitive concepts such as “language”, “working memory”, etc.

While there are clear statistical advantages to combining results across studies, this type of meta-analytic thinking with general cognitive concepts might blur our understanding of structure-function mapping. The ability to map terms like “language” or “working memory” to specific cognitive processes that reside in specific brain regions depends on a) the relationship between the frequency of a specific term in a paper and whether a process related to that term was specifically manipulated in the imaging contrast and b) whether that term has a one-to-one relationship with a specific cognitive process. There is no doubt that there is an issue with structure-function mapping with these very broad terms. The question of where “language” resides in the brain is massively underspecified; “language” is not a single cognitive process and there are clear dissociations between language production and comprehension, between written and spoken language. While it is clear that broad concepts like “language” fail at having a one-to-one relationship with specific cognitive processes, it might appear as though more specific terms, such as “orthography”, may not face the same issue. However, cognitive neuropsychological investigations of individuals with “orthography” impairments in spelling have shown that even the concept of “orthography” involves a series of different, dissociable sub-processes that likely have their own neural instantiation, as they can be separably damaged by stroke (Rapp & Fischer-Baum, 2015). Alternative approaches to meta-analyses of neuroimaging data attempt to address these concerns. For example BrainMap (Laird, Lancaster, & Fox, 2005) allows the user to manually curate the studies to include in the meta-analyses. Even in this tool, however, the information entered is focused more on task manipulations, and less on the cognitive processes involved in this task (see Poldrack & Yarkoni, 2016 for a discussion). The usefulness of these meta-analytic approaches to synthesizing neuroimaging studies is limited by the extent to which we are convinced that the manipulations in the studies being combined map onto the same cognitive process.

This limitation has led Poldrack and others to propose the development of a formal “cognitive ontology” for characterizing mental processes (e.g. Poldrack, 2010; Price & Friston, 2005). Inspired by ontologies utilized by biologists that formally specify the relationship between gene products and gene functions, the idea is that one can formally specify the relationship between mental concepts and mental tasks. For example, one can define a putative cognitive process based on whether it is a part of some other process or a kind of process, and relate specific tasks (with their own experimental conditions and contrasts) to this process. Assuming a fully formed cognitive ontology³, in which every task has a process mapped to it, one could label the results of every imaging study based on what processes are activated for each neuroimaging study. Utilizing the entire neuroimaging literature has been claimed to have multiple benefits for understanding brain-behavior relationships and cognition (see Poldrack and Yarkoni, 2016). One example is isolating cognitive functions. In cognitive psychological and neuroimaging studies, one will often design two experimental conditions that are matched for all but one process. The assumption of pure insertion is made in these studies, where changing one process does not have any additional influence on other cognitive processes. There are many reasons to think that this assumption of pure insertion should not be valid in most cases (Friston et al., 1996). For example, in addition to having a single additional process, a task might change the way attention is deployed or a participant’s motivation. One of the promises of “big data” approaches to neuroimaging is that they can address these limitations of experimental design. For example, Poldrack and Yarkoni (2016)⁴ claim that by simply considering more studies, one can deal with this issue.

In general, there is a concern that claims regarding “big data” approaches for the cognitive sciences may overpromise and underdeliver. It is not clear how one would make the jump from simply having more data from more neuroimaging studies to solving the problem of pure insertion. Although we are not aware of an explicit argument made for how big data would solve this problem, we assume that the logic is as follows. If one has 500 neuroimaging studies involved in the “process” of X, these studies may all vary in other extraneous factors that could violate the assumption of pure insertion at the single study level – i.e. one study may involve additional attentional load, another may somehow involve other higher-order processing, etc. The assumption is that extraneous factors in any single study are just noise that will average out to zero once multiple studies are considered, resulting in cleanly isolating the neural correlates of a specific process. However, it is problematic to assume that the extraneous factors in one study of a cognitive process are really independent from those in a different study of the same process. For example, if the majority of the paradigms used to examine process X also involve some aspect of process Y, the problem of pure insertion will not be solved – regardless of the number of selected studies. More generally, there is the issue of how the process is defined in the cognitive ontology. If the components of the cognitive ontology do not map onto actual cognitive processes, simply increasing the amount of data will not reveal anything substantial about brain-behavior relationships, and/or cognitive processes. To put it another way, it is not clear how meta-analyses of 500 studies of “language”, “working memory”, or even “orthography” will provide substantial advances compared to 25 studies on those same topics. Meta-analyses could provide information if theory-driven. However, if the input is atheoretical and not driven by cognitive processes, the output will be atheoretical and lack specificity.

A shift towards “big data” approaches in cognitive neuroscience may pose an existential threat to the inherently “small data” approaches in single case cognitive neuropsychology. These “big data” methods are innovative and novel, and have the potential to address statistical issues in the field. However, the inferential problems that these “big data” approaches are currently struggling with – specifically how can we isolate the neural substrates of specific cognitive processes – are not novel. They have been the core questions in cognitive neuroscience since its inception. We very much agree with the need for formal specifications of cognitive processes to advance our understanding of the mind and brain. We are just not convinced that “big data” approaches that pull together disparate studies using varying methods, stimuli, etc. will be particularly fruitful as it stands. Instead, we believe that “big data” approaches can only succeed if they are paired with “big theory”. That is, relating brain activity to cognitive constructs in a general way will only allow us to carve nature at the limbs, not the joints, of mental function. It is more likely that a focus on developing testable, well-specified cognitive models will lead to a more efficient utilization of resources and better overall output.

Cognitive neuropsychology and neuroscience – Complementary or in competition?

There are a number of ways that cognitive psychology and neuropsychology can be integrated with cognitive neuroscience. This includes an increased focus on standard cognitive psychological studies that develop cognitive models to “model-based” cognitive neuroscience (e.g. Mack, Preston, & Love, 2013; Palmeri, Love, & Turner, 2017). Importantly, cognitive neuropsychological studies can and should continue to play a role in developing these cognitive models. One of the major contributions of Alfonso Caramazza’s research on cognitive neuropsychology was not just using evidence from brain-damaged individuals to study the mind. Importantly, many of his important works formalized the assumptions made in this method (Badecker & Caramazza, 1985; Caramazza, 1984, 1986; Caramazza & McCloskey, 1988; McCloskey & Caramazza, 1988), with important debates on what we can (and cannot) learn from cognitive neuropsychology (e.g. Caramazza, 1992 versus Kosslyn & Intriligator, 1992). One of the major benefits of this approach was the development of well-specified models of cognitive processes using evidence from brain-damaged individuals (e.g. Caramazza, 1997; Caramazza & Hillis, 1990b; Caramazza & Miceli, 1990; Mahon & Caramazza, 2009; McCloskey, Caramazza, & Basili, 1985; Rapp & Caramazza, 1997; Rapp & Goldrick, 2000). “Big data” approaches hold the promise that is typically expected with new technological advances in any scientific field – the potential for new ideas and even paradigm shifts. However, to truly assess their potential, it is important to be clear and formalize exactly what can (and can not) be found with these new analytical methods.

Finally, the older cognitive neuropsychological approach can be integrated with newer techniques in cognitive neuroscience to understand cognition. This may seem odd to some, given the perceived disconnect between neural data and cognitive theories (e.g. Coltheart, 2006). However, we note that Coltheart was not arguing whether neuroimaging can inform cognitive theories, but whether it had informed cognitive theories. Furthermore, cognitive neuropsychologists have not argued “against the relevance of neuroanatomical or other neurally based observations in constraining cognitive theory…To the contrary…advances in cognitive science and neuroscience will be mutually constraining in the development of a mature cognitive neuroscience” (Caramazza, 1992). For example, changes in BOLD signal in a given region are dependent variables that can be used to infer cognitive function just as reaction time and accuracy are used in cognitive neuropsychological studies (Henson, 2005). Statistical methods for analyzing fMRI data in single, neurologically-intact individuals, whether it be MVPA (Norman, Polyn, Detre, & Haxby, 2006) or the functional localization approach (Fedorenko & Kanwisher, 2009) can be used to examine functional changes in single subjects, including brain-damaged individuals.

For example, neural data can constrain interpretations of behavior. Dilks and colleagues (2007) examined an individual (BL) with an upper left visual field cut who reported objects near this field cut as severely elongated. Neural data established that this field cut was caused, not by damage to primary visual cortex (V1), but damage to the right optic radiations that projected to a fully-intact V1. Functional neuroimaging revealed that stimuli presented in the lower left visual field resulted in activation across the entire right V1, whereas controls only showed activation in upper right V1 for the same stimuli. The extension of activation in V1 was consistent with BL’s reports of vertically stretched objects. Along with providing evidence for plasticity in V1 of adults, neural data provided a clear mechanism for why the observed behavior occurred. Neural data from patients can also provide converging evidence for cognitive theories. Models of word production have shown that information goes through lexical-semantic, followed by lexical-phonological processing, though researchers have debated the time course of when these different processing levels are engaged (see Strijkers & Costa, 2016 for a recent review). Laganaro and colleagues (2011) reported a single individual who took part in an event related potential (ERP) study on single-word spoken picture naming both before and after a stroke (see also Laganaro, Morand, & Schnider, 2009 for a similar approach with a case series). This individual was anomic after stroke, demonstrating a lexical-phonological deficit. Consistent with the behavioral deficit, they also found a large change in ERP components 250–450 msec after picture presentation, a time frame consistent with lexical-phonological processing as had been previously proposed (Indefrey & Levelt, 2004). This study provides evidence using neural data that supports specific models of the time course of lexical access. (For other examples of single-case cognitive neuropsychology and its methods, see Bridge et al., 2013; Mullally, Hassabis, & Maguire, 2012; Price, Crinion, & Friston, 2006; Snow, Goodale, & Culham, 2015; Wolmetz, Poeppel, & Rapp, 2011).

In summary, we argue a detailed understanding of cognitive processes is necessary to understand the mind and brain (see Krakauer, Ghazanfar, Gomez-Marin, MacIver, & Poeppel, 2017 for a similar argument). Big data approaches may provide some information regarding general relationships between brain and behavior, but likely will not advance our understanding of cognition without being rooted in theory. As for cognitive neuropsychology, its value can be assessed by its output – that is, the proof of the pudding is in the eating (Caramazza, 1992). Although cognitive neuropsychology may not be considered “cutting edge”, it has and will continue to provide important contributions towards understanding cognition. Future cognitive neuropsychological work, including strictly behavioral studies and ones that integrate neural data, will continue towards this goal.