Are we ready for real-world neuroscience?

Abstract

Real-world environments are typically dynamic, complex, and multisensory in nature, and require the support of top-down attention and memory mechanisms for us to be able to drive a car, make a shopping list or pour a cup of coffee. Fundamental principles of perception and functional brain organisation have been established by research utilising well-controlled but simplified paradigms with basic stimuli. The last thirty years ushered a revolution in computational power, brain mapping and signal-processing techniques. Drawing on those theoretical and methodological advances, over the years, research has departed more and more from traditional, rigorous, and well-understood paradigms to directly investigate cognitive functions and their underlying brain mechanisms in real-world environments. These investigations typically address the role of one or, more recently, multiple attributes of real-world environments. Fundamental assumptions about perception, attention or brain functional organisation have been challenged - by studies adapting the traditional paradigms to emulate, e.g. the multisensory nature or varying relevance of stimulation or dynamically changing task demands. Here, we present the state-of-the-field within the emerging heterogeneous domain of real-world neuroscience. To be precise, the aim of the present Special Focus is to bring together a variety of the emerging “real-world neuroscientific” approaches. These approaches differ in their principal aims, assumptions or even their definitions of “real-world neuroscience” research. Here, we showcase the commonalities and distinctive features of the different “real-world neuroscience” approaches. To do so, four early-career researchers and the speakers of the Cognitive Neuroscience Society 2017 Meeting symposium under the same title answer questions pertaining to the added value of such approaches in bringing us closer to accurate models of functional brain organisation and cognitive functions.

Introduction

Many fundamental principles of brain organisation, such as hierarchical processing or segregation, have been established by research utilising well-controlled, but simplified, paradigms with basic, artificial stimuli. In contrast, real-world situations, which most experiments intend to model, are typically dynamic, complex, multisensory in nature, and as such rely heavily on a variety of top-down (attentional and otherwise) mechanisms for us to be able to function effectively in everyday life: drive a car, remember our shopping list, etc. Recently, a number of approaches have been developed, largely independently from each other, that aim to bridge traditional neurocognitive paradigms and the demands posed by naturalistic environments. The present Special Focus is the result of an invited symposium under the same title chaired by Pawel Matusz at the Annual Meeting of Cognitive Neuroscience Meeting in March 2017 in San Francisco. The collection of invited manuscripts represents the state-of-the-field within the emerging heterogeneous domain of real-world neuroscience, showcasing the wide range of approaches developed to better understand how the brain works in naturalistic situations.

The ultimate aim of cognitive neuroscience research is to create accurate models of how information processing occurs in everyday situations and how this processing is orchestrated by the brain. The use of simplified paradigms and stimuli has been an incredibly fruitful way to chart the functional organization of the brain, especially in times when little was known about response properties of neurons in different brain areas (Hubel & Wiesel, 1962; Mountcastle, 1957; Evans & Whitfield 1964). These early investigations helped establish some of the fundamental principles of how information is processed by the brain. With time, while such ‘classic’ studies have continued proving their worth throughout the years, new lines of research have been departing further and further away from the traditional paradigms to pursue more complex research questions: discrimination of socially relevant highly similar objects (e.g. Haxby et al. 1996), facilitation of the processing of task-relevant stimuli in multi-stimulus settings (e.g. Moran & Desimone 1985), or neurocognitive processes gauged by stimuli engaging multiple senses at once instead of a single one (e.g. Meredith & Stein 1983; Stein et al. 1988).

Fundamental assumptions regarding both functional brain organization and cognitive processes have been substantiated by these early neuroscientific studies. However, further important novel insights have been afforded by careful, parametric adaptations of the above-mentioned traditional paradigms, which varied task-relevance, social value or the (multi)sensory nature of stimulation. Nowadays, a variety of approaches have emerged, with some departing far from well-controlled experimental paradigms in the pursuit of better understanding of neurocognitive processes in naturalistic settings. A pressing question motivated by studies conducted within these lines of research is: Are we now at the stage where we can safely abandon laboratory-based investigations?

The different approaches falling under the umbrella of “real-world neuroscience” have developed drawing on theoretical advances but also, to differing degrees, on the ongoing advances in signal-processing techniques, computational power and/or brain mapping tools. This fact is important inasmuch as there is a heated debate regarding the extent to which scrutinising neurocognitive functions with multiple imaging methods and/or sophisticated techniques of signal analysis alone can provide us with any clue as to the underlying mechanisms. Reservations in the neuroscientific community regarding many of the novel, technology-heavy approaches are clearly visible in voices of prominent researchers (e.g., Krakauer et al. 2017; see also a provocative study by Jonas and Kording (2017)).

The aim of this Special Focus is to bring together, contrast, and start creating synergies across the variety of the existing “real-world neuroscientific” approaches. These approaches follow distinct definitions of “real-world neuroscience research” and address different shortcomings of existing, “traditional” neurocognitive research. In the “The issue” section below, we describe the main findings and innovations reported in the invited papers. In the subsequent, “Q & A” section, four early-career researchers and the speakers of the CNS 2017 symposium, Suzanne Dikker, Alex Huth, Pawel Matusz, and Catherine Perrodin answer questions, such as those pertaining to most exciting insights offered by studying brain/mind functioning in more naturalistic settings, the readiness of the field for a paradigm shift, or future directions of the field as a whole. Their responses aim to showcase both distinctive features of the different “real-world neuroscience” approaches as well as their commonalities.

One such commonality may be conceptualisation of neuroscientific investigation as a three-stage cycle akin to the “exploration-confirmation” cycle traditionally proposed for the scientific method (see Fig. 1). In this cyclical model and contrasting with real-world research, “classic” neuroscientific research (Fig.1A) offers piecemeal insights into naturalistic perception and action, by testing detailed hypotheses regarding specific aspects of a task or situation, with maximised statistical power (enabled by simplified experimental designs). For this purpose, it manipulates a minimal number of factors in paradigms that isolate the process of interest (process-specific tasks), e.g., one type of stimuli or task, with participants tested with non-mobile brain imaging tools (in the case of non-human animals, using head-fixed or anaesthetised subjects). The highly-detailed hypotheses generated and falsified by these approaches have been critical for establishing many fundamental principles orchestrating neurocognitive functions underlying perception and action.

Figure 1.

A schematic illustration of a three-stage cyclical model of the advantages and limitations in investigating neurocognitive functions of fully naturalistic studies and more traditional, laboratory-based approaches. A. “Classic” neuroscientific research, conducted inside the laboratory (e.g. involving participants performing simple tasks involving a single type of stimuli, while their brain responses are recorded with MRI or EEG), provide the opportunity to test detailed hypotheses related to particular aspects of a situation/task. These approaches have maximal control over stimulation and the testing environment, typically at the expense of ecological validity of the tested processes. B. Naturalistic laboratory research provides a closer approximation of natural perception and action while maintaining a certain level of stimulus and environment control. These approaches test models of neurocognitive functions in settings resembling everyday situations by using traditional, process-specific paradigms, but adapted so that they emulate variabilities present in real-world situations (e.g. testing non-human animals inside laboratory cages where the animals can move freely; in the case of research in humans, brain responses are still recorded in the laboratory, using MRI, EEG, etc. but both hypothesis- and data-driven approaches are often employed), but the paradigms involve systemic manipulations of e.g. stimulus formats (e.g. visual, auditory, audiovisual), stimulus relevance and/or task demands. C. Research conducted in veridical real-world situations (e.g. in classrooms, or with naturalistic stimuli, such as movies) enable scientists to directly test the extent to which laboratory-generated hypotheses and theories of perception and action hold in the real world, and may reveal novel factors and mechanisms influencing these functions in everyday environments. The latter can then be examined piecemeal with “classic” neuroscientific approaches (A) or tested more holistically in controlled but more naturalistic settings with naturalistic approaches (B). Note: Images depicting MRI and EEG (inside the laboratory and the classroom) setups as well as that of mice in a laboratory cage have been provided due to the courtesy of (from top left to bottom right): Alex Huth, Nora Turoman, Diane Quinn (© 2015 Trevor Day School), and Bridgette Archer.

At an intermediate point on the spectrum ranging from approaches with maximal parametric control to those with maximal behavioural relevance, real-world-like laboratory approaches (Fig.1B) test models of neurocognitive functions in settings resembling everyday situations. Specifically, they constrain the range of potential processes present during the experiment by also using process-specific tasks but tasks adapted so that the “noise” characterising real-world situations is still present, albeit in a reduced, i.e., more controlled, fashion. That is, such real-world-like approaches emulate typically two or multiple important known variabilities present in natural situations, i.e., those in stimulus task-relevance, complexity, familiarity and/or format (visual, auditory, tactile, and their multisensory pairings), those in task demands and/or those in capabilities and experience of the observers (see the Q&A section for more details). Here, hypothesis-driven analyses help to falsify predictions regarding involvement of specific neural mechanisms in given naturalistic functions, while data-driven approaches help reveal new insights into the governing mechanisms and/or moderating factors. The presence and/or importance of these factors in naturalistic environments is then verified in studies conducted in fully naturalistic real-world studies (Fig.1C), which in turn generate new questions. These can be then examined either individually in traditional neuroscientific studies (Fig.1A) and/or tested more holistically in the naturalistic studies studies, in settings more closely approximating the real world (Fig.1B). In this model, research conducted in veridical real-world situations (e.g. EEG studies in public places, a large group of mice interacting freely in a natural ecosystem) and/or with unconstrained natural stimuli (e.g. genuine films or narrated stories; Fig.1C) tests the extent to which laboratory-generated hypotheses and models of naturalistic perception and (social) action generalise outside of traditional laboratory investigations. As these real-world approaches often heavily rely on data-driven analyses, they have particular potential to reveal novel factors and mechanisms orchestrating cognition and behaviour in the real world.

“Real-world neuroscience” state-of-the-field: The issue

The papers included in this Special Focus represent - in a necessarily limited fashion - the wide range of exciting approaches and ideas within the area of real-world neuroscientific investigations. Some of these studies provide novel insights into the role of most stimulus- and context-related factors in influencing object detection and discrimination as well as the associated functional brain organisation in naturalistic environments. Murray et al. report that moment-to-moment detection of dynamic and stationary stimuli is less variable in multisensory than in unisensory contexts. They used reference-independent analyses of global features of the electrical scalp field (electrical neuroimaging; e.g., Murray et al. 2008) to reveal that this performance improvement is related to distinct patterns of functional connectivity between nodes of a distributed occipital-parietal-frontal network. In turn, purely visual interactive processes across real-world scenes and objects are scrutinised in a multivariate pattern analysis fMRI study from Brandman and Peelen. The authors show that indoor and outdoor scenes are both better recognised when accompanied by an intact object and that these benefits from these interactions across categorical visual information drive representations within the parahippocampal place area but not the occipital place area. This study is complemented by a contribution from Turoman et al. who highlight how traditional visual selective-attention paradigms can be adapted to better understand the brain mechanisms underlying the attentional selection of visual objects controlled by top-down multisensory task-determined templates. The N2pc component, a widely-accepted event-related potential (ERP) marker of attentional object selection, triggered by visual stimuli was attenuated across purely visual versus multisensory search tasks. Electrical neuroimaging analyses (see above) revealed that in contexts of partial match with the multisensory template, the visually-induced N2pc can be generated by a different brain network across full and partial match with the template, rather than by gain-control modulations of activity of the same brain network (as typically suggested for N2pc). Together, these findings showcase the improved neural processing efficiency as a hallmark of information processing in naturalistic environments, where integrative processes (both multisensory and unisensory) are common; they also highlight the importance of top-down, semantic and task-dependent, processes that likely simultaneously influence information processing (i.e., object recognition and attentional selection) in naturalistic settings.

Other studies in this Special Focus showcase the importance of development and individual’s experience in shaping cognitive functions and their brain underpinnings. Wu et al. demonstrate how the already mentioned N2pc component can be used to study the development of top-down attentional control based on object templates defined by abstract object categories (see Turoman et al. above, for use of N2pc to study top-down object templates defined by perceptual features). Specifically, the authors present combined behavioural and ERP results that N2pc can be used to verify whether a given individual groups certain items as belonging into the same abstract category. Building on other well-established ERP correlates of visual attentional control processes, Föcker et al. show that extensive experience with action video games is linked to changes in early attentional processes reflected by the anterior N1 component, which suggests that gaming experience equips individuals with superior abilities to direct to and maintain the focus of spatial attention in the attended location(s). These results highlight one of possible mechanisms whereby an individual’s experience, such as that in gaming, improves higher-level cognitive functions, such as visuospatial attention.

Taking neuroscientific investigations into veridical naturalistic environments, Bevilacqua et al. use portable electroencephalography (EEG) headsets to record brain activity from a group of high school students and their teacher during their regular classes. They demonstrate how teacher likability and test scores vary as a function of inter-brain synchrony (coherence) among the group of students, and between students and their teacher. Lastly, going fully beyond the canon of cognitive neuroscience research, but very much within the aims of this special issue, Altikulaç et al. shed light on the importance of information available in the media regarding “the adolescent brain” as sources of control over cognitive and emotional control processes in adolescents. The authors provide novel data pointing to the wide-ranging impact of framing such neurocognitive findings by the media - in a positive (focus on creativity, flexibility, etc.) versus in a negative fashion (focus on increased risk taking, low capacity to plan, etc.) - on adolescents’ behaviour ranging from risk taking to response to failure.

“Real-world neuroscience” state-of-the-field: Q & A

What does real world neuroscience mean to you?

AH: I think it is difficult to answer this question without some historical perspective. Throughout the 20th century, psychologists sought to make their field more scientific and quantitative by constructing elaborate, tightly controlled laboratory experiments. Scientifically, this was a good move! But this reductionism also has a clear downside. That is, restricting one’s scope of inquiry to tightly controlled experiments makes it difficult or impossible to study most human behavior. The modern alternative to this 20th century mindset is to perform less-controlled experiments using ethologically relevant, natural stimuli. I believe that this type of experiment can go much further at probing the neurobiological bases of complex behaviors like language and social interaction. And I believe that this type of experiment has only become possible recently, thanks to advances in computational power, data acquisition, and data storage. Using these factors we have less need to control for confounding factors (like our 20th century forebears) because we can model them computationally. This revolution is what I consider real-world neuroscience.

SD: My collaborators and I take the notion of “real world neuroscience” quite literally, i.e., conducting neuroscience outside of laboratory context, in the real world. We do not, however, predict that real-world neuroscience will ever render laboratory-based research obsolete. My answer to the question posed in the Introduction “Are we now at the stage where we can safely abandon laboratory-based investigations?” is an emphatic and decisive “no.” Instead, we think of real-world research as a complementary approach that can inform, enrich, and inspire laboratory research, and vice versa, as illustrated in Figure 1. The advance of real-world neuroscience research enables researchers to test the long-standing assumption that the laboratory serves as a valid model for real-world human behavior, to test hypotheses that cannot be addressed in the absence of face-to-face interaction, to address hardware challenges that can help advance technological innovation, and to gather data that can inspire and inform subsequent laboratory experimentation.

PM: In my opinion, the two approaches particularly useful for clarifying the neurocognitive architecture supporting object recognition and social interactions in everyday situations are naturalistic laboratory studies (Fig.1B) and studies conducted in veridical natural situations (Fig.1C). “Classic” approaches (Fig.1A) - with their focus on developing and testing detailed hypotheses about processes operating in specific contexts - have clarified the relative importance and the neural mechanisms governing space-, feature- and object-related top-down goal-driven processes controlling our attention to visual and auditory stimuli. However, many cognitive functions and behaviours are likely emergent, i.e., they cannot be understood by analyzing the underlying neural circuits alone (bird flocking is a great example here). Indeed, the neurocognitive architecture governing attentional control might dramatically different while performing even the most trivial everyday actions. For example, how we are attending while watching a movie might be fundamentally organized by the emotional value of the movie events, what our viewing companions are paying attention to, and/or the strength of our habit of constantly switching attention between the movie, chatting/ social media and eating or taking out the washing to dry. Fully naturalistic studies (Fig.1C) can help verify the relative importance of such everyday demands on our information processing as well as reveal others. In turn, naturalistic laboratory experiments (Fig.1B) can falsify hypotheses and models regarding the neurocognitive underpinnings of attending selectively in such situations by systematically manipulating the spatial location, emotional value, sensory modality, emotional value and social cues related to the targets and distractors. Research has primarily focused on manipulating systematically one, perhaps two of these demands. More recently, approaches adapting traditional paradigms to systematically manipulate stimulus task-relevance, task demands and/or skills of observers in multisensory settings has provided important novel insights into the extant models of brain organization, perception, language or cognitive development (Alsius et al. 2005; Laurienti et al. 2006; Iordanescu et al. 2008; Lewkowicz & Ghazanfar 2009; Cappe et al. 2010; Matusz & Eimer 2011, 2013; Reich et al. 2012; Alsius & Munhall 2013; Matusz et al. 2015a; Parise & Ernst 2017). Thus, neuroscientific studies that test hypotheses and models by combining process-specific tasks together with the paradigm adaptations that emulate the demands of natural environments and new technologies could provide one useful way of falsifying models of brain mechanisms orchestrating real-world cognition and behaviour (e.g. filial imprinting; Vallortigara et al. 2005). A particularly promising avenue for combining unconstrained behavior and well-controlled contexts might be the use of virtual or augmented reality setups (e.g., Rizzo et al. 2000; Murray et al. 2015; Vedamurthy et al. 2016).

CP: To me, “real-world neuroscience” means adopting a neuroethological approach to the study of brain function - in the laboratory: that is, investigating neuronal mechanisms in awake behaving animals, in the context of ethologically relevant situations and/or using complex natural stimuli. Experimental settings that preserve at least the key environmental features a given species’ brain has evolved to use allow tapping into an animal’s natural behaviour, and are instrumental in revealing the neuronal substrates of ecologically-valid perception and action. The challenge lies in managing a balance between the demands of a controlled, parametric environment necessary for the neuronal-level interrogation of brain circuits, and the degree of behavioural relevance of a laboratory-based approximation of the real world (e.g. Juavinett et al. 2018).

What should people know about current paradigms and their limits and how does your work fit into this “new future”?

AH: The fundamental issue is that the brain is not a linear system. If the brain were linear, then we could reasonably expect that results obtained from simplified, controlled stimuli would generalize to natural stimuli. Let’s suppose we record V1 neurons responses to every possible image that has one single active pixel. If the brain were linear, these responses should predict how the neurons respond to real images. However, already in V1 some responses are nonlinear, as they depend on motion (i.e., complex cells). Outside of V1, neuron responses to single pixels cannot explain the activity of neurons that respond specifically to faces (Kanwisher et al., 1997; Chang & Tsao, 2017). Natural stimuli have been demonstrated to be advantageous here (e.g. David, Vinje, & Gallant 2001).

Furthermore, in many fields, there is little effort to test whether results from controlled studies generalize to real world situations. Every experimentalist should ask themselves: Does the result of this experiment imply anything about cognition/behavior of this organism in the real world? Findings from controlled studies are naturally limited to the hypotheses specified in the experimental design. Studying how the brain responds to natural, ethological stimuli solves these problems. However, it creates other problems: Many things are correlated in natural stimuli, so hypotheses can be difficult to distinguish. This necessitates collecting more data and using more careful statistical methodology.

SD: To date, the study of the human mind/brain through behavioral, neurobiological, and computational techniques has relied on one very fundamental assumption: Laboratory-based research provides foundational insights into how our brains process information on an everyday basis. However, we rarely test this assumption by conducting studies in the actual real world. Some research questions, such as those involving face-to-face interaction, are virtually impossible to investigate in the laboratory: The study of real-world social exchanges has even been dubbed the “dark matter of social neuroscience” (Schilbach et al., 2013). How similar are the brain processes of a 20-year old college student who is pressing buttons during a visual search task while wearing noise-canceling headphones inside an experiment booth and those of a 45-year working parent who is late for work and looking for her glasses on a cluttered breakfast table, while her 5-year-old child is pulling on her trousers, her 2-year old is smearing egg yolk all over the table while dangerously balancing on his high chair, and the woman’s husband is trying to discuss the day with her?

At the same time, this example illustrates why neuroscience research has been mostly confined to the laboratory. Naturalistic environments are messy, noisy places in which it is virtually impossible to obtain good experimental control. Also, there are obvious practical limitations: You cannot place cutting-edge lab equipment such as MRI or MEG machines into classrooms (or kitchens). While simpler, lower-grade equipment has become increasingly accessible (e.g., Debener et al., 2012), most of this technology is still limited in several respects (e.g., data quality and lack of localization options). As such, it is unrealistic to expect the same level of data quality and experimental control from real-world neuroscience research as we demand from laboratory experiments, and research questions and expectations should be adapted accordingly.

PM: The current initiatives involving collecting data from larger samples and sharing data across different groups are applaudable, and certainly can help to make cognitive neuroscience more replicable (Poldrack et al. 2017). However, as I have already mentioned, the mechanisms governing cognitive functions and functional brain organization in natural situations will remain elusive if the employed paradigms continue not to emulate the demands typical for these situations. The existing models are based on research that adapted the early simplistic paradigms to emulate one or another attribute of naturalistic environments, e.g., their multistimulus nature or the role of memory in object processing (e.g., Folk et al. 1992; Horwitz et al. 1998; Gobbini & Haxby 2007; Hickey et al. 2009). However, it remains largely unclear whether/how well these unisensory (visual, auditory) models generalize to everyday situations that are profoundly multisensory in nature. It is increasingly recognized that neural representations of objects are inherently multisensory (e.g., Reich et al. 2012; Murray et al. 2016a). Yet, influences of multisensory processes on brain and cognitive processes are often involuntary; and due to the non-linear nature of the underlying integrative mechanisms, these influences cannot be predicted from the unisensory responses alone (reviewed in e.g. Murray & Wallace 2012; De Meo et al. 2015).

Multisensory paradigms, especially those carefully manipulating stimulus formats (unisensory versus multisensory, to help assess the presence/strength of multisensory processes), task relevance and/or demands, are more emblematic of object processing in the real world. My collaborators and I have shown that traditional adult selective attention and memory paradigms readily lend themselves to adaptations to multisensory settings. Drawing on the rigor of these paradigms, the well-established behavioral measures of specific processes and the breadth of associated literature, we have been making progress in bridging the traditional models of perception, attentional control and learning/memory with research on multisensory processing. As a result, we have clarified the role of multisensory processes as bottom-up and top-down processes controlling object attention and memory, and how these influences depend on the individual’s age and experiences. Where possible, we have also utilized the rich spatio-temporal resolution afforded by high-density EEG and electrical neuroimaging to refine the existing models of functional organization of object processing and top-down attentional control (e.g., Matusz & Eimer, 2011, 2013; Matusz et al. 2015b; Thelen et al. 2014; Turoman et al., this issue). We have shown recently that EEG might be particularly useful to image brain activity in veridical real-world environments, especially where task compliance is difficult or impossible (e.g., Maitre et al. 2017; see below).

CP: Historically, insights into the neuronal-level underpinnings of perception have been provided by studying how cortical neurons in anesthetized animals process and encode sensory features present in simple artificial stimuli. This approach has successfully uncovered fundamental principles governing the functional organization of sensory systems, including the diversity of topographic maps in the visual and auditory cortices (Merzenich, Knight, & Roth, 1975; Ohki et al., 2006). The brain, however, typically functions as a nonlinear system, with complex stimuli processed and perceived holistically. Similarly, it has proved challenging to predict neuronal responses to natural stimuli from those to basic synthetic components in isolation: activity patterns elicited by faces or communication sounds are more successfully estimated from responses to high-dimensional features directly extracted from the relevant natural stimuli (Chang & Tsao, 2017; Machens, Wehr, & Zador, 2004; Theunissen, Sen, & Doupe, 2000). In addition, traditional laboratory-based task contexts, such as those involving stimuli presented to animals under the influence of anaesthetics, using artificial stimulus categories, or imposing arbitrary associations between certain stimuli and a reward, likely engage different mechanisms than when the animal responds to a naturalistic stimulus in an ethologically-relevant situation.

The goal of my research is to uncover the neuronal substrates enabling mammalian sensory communication. Specifically, I study how groups of neurons in the brain of the listener encode social information found in complex sound patterns. To probe brain processes involved in analyzing real-world, meaningful social signals, my colleagues and I have, by necessity, moved away from simple auditory and visual stimuli towards conspecific vocalizations and their corresponding dynamic facial expressions. Our work has helped clarify the neuronal representation of communication signals in the temporal lobe of awake behaving nonhuman primates; in an illustration of studying ethologically-relevant sensory processing in a laboratory setting, we collected single-unit and oscillatory neuronal responses to conspecific vocalizations and their associated facial expressions (Perrodin, Kayser, Logothetis, & Petkov, 2014, 2015). This type of approach is a key first step towards directly probing how the brain encodes social communication signals at the neuronal level when perceiving other individuals in a naturalistic, multisensory context (Perrodin, Kayser, Abel, Logothetis, & Petkov, 2015).

What do we now understand about the brain via real-world studies that we did not previously?

AH: One of the most dominant ideas in the neuroscience of language (and perhaps in the entirety of human neuroscience) is that language processing occurs mainly in the left cerebral hemisphere (in most people). That idea is supported by a wealth of neuropsychological and neuroimaging research, most of which utilized isolated words or other non-natural language tasks. However, more recent neuroimaging studies that used natural, continuous language (Lerner et al., 2011; Honey et al., 2012; Regev et al., 2013; Silbert et al., 2014; Huth et al., 2016; de Heer et al., 2017) seem to suggest that the right hemisphere is as involved in language processing (and production) as the left. Areas in the right hemisphere seem to play an equal (or, at least, substantial) role in the processing of every aspect of language, including phonetics, syntax, and semantics. It is not yet known why these effects are only seen (or are seen more readily) using natural language stimuli, but that is an area for future exploration. This up-ending of the traditional theory of left-lateralization for language clearly shows the need for and importance of studying how the human brain responds to natural, ethologically relevant stimuli.

SD: I can speak most specifically on conducting neuroscience research in the classroom. As real-world neuroscience “laboratories”, classrooms are highly social environments but they still afford some experimental control: Kids are used to engaging in short sessions of different ”conditions” (listening to a lecture, watching a video, engaging in a group discussion) while sitting still and being focused. This semi-structured nature of the classroom makes it possible to explore how social, contextual, and individual variables influence the extent to which the brain activity of a class of students becomes synchronized. In our work specifically (Dikker et al. 2017, and Bevilacqua et al., this issue), students’ brain-to-brain synchrony during classroom activities was correlated with how much they liked a given class and each other/the teacher. This suggested that neural entrainment/synchrony can stem from different sources: the nature of what we see or hear, how much we like it, how focused we generally are, and, perhaps most interestingly, the social dynamics of the situation we are in. These findings corroborate other evidence suggesting that social interaction matters to how we experience the world, even if what we do after engaging with someone is not immediately social in nature (e.g., watching a movie). Some other findings would have been difficult, if not impossible, to obtain in a laboratory environment. For example, brain activity of pairs of students who had directly interacted with each other before class was more in sync during class, and this brain synchrony was correlated with students’ social closeness. In other words, friends’ brainwaves were more synchronized but only if they interacted with each other. In another project, this time in collaboration with artists, we managed to record EEG data from a large sample of people from a wide age range of ages and backgrounds, in museums across the world. This allowed us to investigate how the brain supports unconstrained face-to-face interaction, building on laboratory verbal communication studies that have used either more constrained tasks or lack direct face-to-face interaction (Stephens et al., 2010; Dikker et al., 2014; Cohen et al., 2017; Dumas et al., 2010; Sanger, Müller & Lindenberger, 2012; see Hasson et al. 2012; Babiloni et al., 2013 for reviews; Jiang et al., 2015, Nozawa et al., 2016, and Liu et al., 2016 for face-to-face verbal communication studies).

PM: To paraphrase the provocative title of Asif Ghazanfar and Charles Schroeder’s (2006) paper, large swaths of the neocortex (and subcortex) have the ability to integrate information stimulating multiple senses. Crossmodal convergence as well as, increasingly, multisensory processes are assessed in terms of their utility in the clinics for improving sensory and cognitive deficits (e.g., Amedi et al. 2007; Striem-Amit et al. 2012; Murray et al. 2015; Tinga et al. 2016; Vercillo et al. 2017; Bottari et al. 2018). I believe that the real potential lies here in scrutinizing the interplay between top-down attentional control, object processes, and brain development/plasticity, as it has been done in the purely visual domain (e.g., Astle & Scerif 2011; Gazzaley & Nobre 2012). The recent efforts to integrate the breadth of multisensory research spanning different populations, paradigms and imaging methods, including those by my collaborators and I, have been quite fruitful.

First, integrative processes can occur at “feedforward” (<100ms post-onset) stages of brain processing, within “sensory specific” cortices, such as V1 (reviewed in e.g. Atteveldt et al. 2014; De Meo et al. 2015; Murray et al. 2016a; Deroy et al. 2016). As a result, co-occurring crossmodal stimuli are often easier to detect, perceive and/or attend to, compared to unisensory stimuli, irrespective of the observer’s goals (e.g. Giard & Peronnet 1999; Cappe et al. 2010; Matusz & Eimer 2011; Sarmiento et al. 2016). Second, in real-world-like, multi-stimulus settings, semantic multisensory processes might be especially important for implicit benefits in recognition of and attention allocation to naturalistic objects (e.g., Laurienti et al. 2004; Murray et al. 2004; Iordanescu et al. 2008; Alsius & Munhall 2013; Matusz et al. 2015a, 2015b; reviewed in ten Oever et al. 2016; Muray et al. 2016b; Matusz et al. 2017). Third, as already hinted, multisensory research reveals new insights about attentional control and its functional organization. Multisensory objects are selected involuntarily, independently of unisensory task demands, with multisensory top-down templates controlling responses in “sensory specific” cortices (Matusz & Eimer 2011, 2013; Matusz et al. 2015a; Turoman et al. this issue). Fourth, multisensory research extends some of the exciting findings from the visual domain that point to the central role of object familiarity/expertise in determining the efficacy of top-down attentional control (e.g., Wu et al. 2015, 2016), by showing paradoxical, potentially protective effects of age on attentional control in real-world situations (e.g. Matusz et al. 2015a).

What to me seems to hold the most promise in education and clinics - two highly pertinent real-world settings - is identifying robust behavioral measures of cognitive processes of interest, and use them to understand changes therein associated with age, neurodegeneration, but also rehabilitation. Here, easy to use and low-cost methods of brain imaging, such as EEG, can be particularly useful in understanding the underlying brain mechanisms.

CP: In a series of studies, my colleagues and I targeted neurophysiological recordings to a higher-order voice-sensitive area in the superior temporal lobe of awake behaving rhesus macaques. We identified a population of neurons in the anterior temporal plane characterized by a categorical response preference to conspecific vocal signals over other complex natural sounds. These functionally specialized “voice cells” were highly-stimulus selective, and each responded to a different subset of the presented voices (Perrodin, Kayser, Logothetis, & Petkov, 2011). Interestingly, neurons in the voice area were sensitive to specific communicative features in the vocal sounds, such as either call type or caller identity, suggesting a form of functional segregation in the neuronal-level encoding of different aspects of vocal processing (Perrodin et al., 2014). To understand the processing of acoustic communication in its frequently multisensory context, we then combined conspecific vocalizations with dynamic facial expressions. This approach revealed that an auditory-only characterization of response properties only provides part of the picture: Almost half of neurons in the voice area and in the superior-temporal sulcus showed spiking responses to voices that were modulated by the simultaneous presentation of faces (Perrodin et al., 2014). Visual input influenced auditory responses along the temporal lobe with varying degrees of sensitivity to the congruency in the voice-face pair. In addition, the direction of visual modulation was correlated with the temporal relationship between the onset of the facial expression and the vocal sound (Perrodin, Kayser, Logothetis, et al., 2015).

Together, these findings emphasize how key response properties of many cortical neurons, especially in hierarchically higher-level areas, are best captured when using behaviourally-relevant, complex stimuli, and cannot be easily predicted using low-level stimuli in isolation or when considering only the dominant sensory modality. More generally, our research also corroborates a growing body of evidence indicating that much of cortical processing is multisensory, and that information streams from different senses are not processed in isolation from each other.

What questions do you most want to see answered by real-world studies?

AH: Every question about sensation or cognition could benefit from examining the brain under natural circumstances. However, there are certainly some areas where the tools of classical neuroscience are less appropriate, and real-world neuroscience could be of more use. In particular, social interaction (including language!) is one cognitive domain where natural paradigms seem critical for understanding.

SD: I have answered this question more elaborately above and below. Some of these include: How does the brain support real-world dynamic social interactions? Is laboratory neuroscience a valid model based on which inferences can be drawn regarding neural mechanisms underlying our everyday interactions with our environment?

PM: Neuroscientific research conducted in veridical real-world and in naturalistic settings has the potential to address such pressing questions as how the brain copes with noisy environments and the circumstances and mechanisms that enable objects/events to be processed efficiently. Here, particularly needed are careful investigations into maturational and experience-based changes in processes controlling attending towards multisensory objects in healthy and atypically developing populations. Paradigms emulating information processing demands characteristic of the real world, combined with robust methods of measuring brain activity (e.g. electrical neuroimaging; Murray et al. 2008), are increasingly shown to verify and enrich the existing models of sensory processing and its interactions with top-down control as they occur in everyday situations. Immersive environments, such as those involving tablet games and/or virtual realities, could be particularly viable to emulate the demands of these situations, including varying task demands and stimulus relevance, with much of the stimulation being multisensory in nature. There is a notable disparity between findings in both healthy and atypical populations with respect to the so-called far transfer, i.e., improvements of untrained cognitive functions, across the “cognitive training” regimes and certain types of video games (cf. Franceschini et al. 2014; Astle et al. 2015; Vedamurthy et al. 2016; Melby-Lervåg et al. 2016; Merkley et al. in press). Notably, immersive environments could address the problem of the constrained nature of studied behaviors that is characteristic of in-lab experiments. There is real potential in creating synergies across advances in neurorehabilitation and technology and those pertaining to the theoretical and methodological achievements of both experimental psychology and cognitive neuroscience. Much of the current work conducted by my collaborators and I focuses on building these synergies.

CP: How does our brain enable us to communicate with each other? What are the neuronal mechanisms for extracting, encoding, perceiving and using social cues from the auditory or multisensory signals emitted by another individual?

Do you think classical lab studies will be a thing of the past? How best to bridge the new methods?

AH: A well-controlled study with clear null and alternate hypotheses is always going to be the best way to test a specific hypothesis. The problem is that, except in very few cases, we don’t know which hypotheses to test. The space of hypotheses is too big, and we know too little about the system. This is why we need real-world/natural experiments: to show us the general shape of the hypothesis space efficiently, instead of shooting in the dark with controlled experiments.

SD: As I already pointed out above, and as is illustrated in Figure 1, we do not believe that laboratory studies are going to be a thing of the past. Instead, we think of real-world research as a complementary approach that can inform, enrich, and inspire laboratory research, and vice versa. The advance of real-world neuroscience research enables researchers to test the longstanding assumption that the laboratory serves as a valid model for real-world human behavior, to test hypotheses that cannot be addressed in the absence of face-to-face interaction, to address hardware challenges that can help advance technological innovation, and to gather data that can inspire and inform subsequent laboratory experimentation.

PM: To reiterate, rigorous paradigms emulating information processing demands characteristic of everyday situations, combined with robust brain imaging methods are particularly well suited in testing models of neurocognitive functions as they occur in the real world. One way in which to combine them with those more unconstrained neuroscientific investigations in veridical naturalistic settings is in a form of the classic “exploration-confirmation” scientific investigation cycle, a version of adapted towards real-world neuroscientific investigations I and other Symposium’s speakers provided here in Figure 1. Overall, I believe that the most promising approaches will be those that will combine the main advantages of the two directions: the use of tested behavioral indices of cognitive processes of interest, and weaker constraints on available behaviors.

CP: Laboratory-based work is more relevant than ever, considering the democratization of recent technological developments that allow more powerful monitoring and perturbation of neuronal circuits in awake, freely-moving animals (e.g. ultralight and/or wireless hardware for recording neuronal ensembles, high-density silicon probes for monitoring hundreds of neurons simultaneously, optogenetic and minimally invasive molecular-genetic tools for manipulating neuronal activity). These techniques can uniquely be combined with natural stimuli in ecologically valid behavioural assays to answer fundamental questions about how brains process the real world. However this abundance of methods should not occlude the importance of neuroethological approaches to understanding brain function (Krakauer et al. 2017).

What do you want to be the take-home message of the symposium for the public?

AH: The time has come for real-world neuroscience. We know how to design natural experiments and we have the technology to process the data. Perhaps the most difficult element of the natural experiment is how to interpret the results. That is where I believe we will see the most advancement in coming years.

SD: Conducting real-world neuroscience research “in the wild” is possible and can provide rich datasets that help elucidate how the brain supports real-world dynamic social interactions. While we are clearly at the dawn of such efforts, the proofs of concept that we now have are important, if only in light of the fact that the neuroscience research community remains (rightfully) skeptical. As a side note, I want to have mentioned that directly involving the general public in research is mutually beneficial: As researchers, we get access to highly motivated participants, while our subjects become acquainted, in a hands-on and intuitive way, with neuroscience research and the scientific process.

PM: The current initiatives involving collecting larger amounts of data certainly can help make cognitive neuroscience more replicable (Poldrack et al. 2017). However, without appropriate paradigms, i.e., those emulating the information processing demands characteristic of natural environments, the mechanisms governing ognitive functions and functional brain organization in such settings will remain at best incomplete. Particularly promising, especially for education and clinics, are those approaches to testing models of cognitive functions that carefully chart possible synergies across advances in experimental psychology and cognitive neuroscience and those in technology.

CP: Brains look the way they do for a reason - they have been optimized over the course of evolution to best solve the specific set of environmental/social challenges that each species faces in their ecological niche. In order to ultimately understand the brain, we need to complement the existing knowledge, which has been acquired using highly artificial, simple stimuli and situations, with real-world approaches that study brains in the ethologically relevant situations they are designed to handle. We are now ideally placed to harness recent technological and analytical developments in multidisciplinary research that can embrace the complexity of the real world.