Mobile cognition: imaging the human brain in the ‘real world’

Abstract

Cognitive neuroscience studies in humans have enabled decades of impactful discoveries but have primarily been limited to recording the brain activity of immobile participants in a laboratory setting. In recent years, advances in neuroimaging technologies have enabled recordings of human brain activity to be obtained during freely moving behaviors in the real world. Here, we propose that these mobile neuroimaging methods can provide unique insights into the neural mechanisms of human cognition and contribute to the development of novel treatments for neurological and psychiatric disorders. We further discuss the challenges associated with studying naturalistic human behaviors in complex real-world settings, as well as strategies for overcoming them. We conclude that mobile neuroimaging methods have the potential to bring about a new era of cognitive neuroscience, in which neural mechanisms can be studied with increased ecological validity and with the ability to address questions about natural behavior and cognitive processes in humans engaged in dynamic real-world experiences.

Introduction

One of the main goals of cognitive neuroscience is to understand how the brain supports natural human behavior and cognition and, ultimately, to be able to detect and treat malfunctions in the underlying neural systems. Research over several decades has provided invaluable insight into the neural mechanisms that support human behavior and cognition, using intelligently designed experimental tasks that are completed while brain activity is measured using a variety of neuroimaging techniques¹. Studies of human cognition in the real-world (without brain recordings) have been able to investigate naturalistic human behaviors using tools such as wearable sensors (enabling, for example, eye-tracking or measurements of heart rate or electrodermal activity), smartphone apps, location tracking, ecological assessments, photo or video capture and other ecologically valid methods for behavioral data collection in the ‘wild’^2–6. However, because traditional human neuroimaging methods require bulky equipment and participants to remain motionless during recordings, these studies lacked the ability to simultaneously record high-quality signals from the brain. Human neuroimaging studies have thus taken place predominantly in tightly-controlled laboratory settings using experimental tasks and stimuli that are not necessarily reflective of dynamic and complex real-world scenarios⁷. At the same time, neuroimaging studies in non-human species provide evidence that spontaneous, natural behavior and the degree of naturalism of stimuli have a strong impact on neural activity throughout the brain^8–18. Consequently, it remains unclear to what extent our current knowledge about the neural substrate (that is, the activity patterns of single neurons or neuron populations) supporting human cognition and behavior generalizes from laboratory studies to real-world experience. Moreover, large gaps in knowledge remain with regards to questions that cannot be adequately addressed with brain recordings during immobility, such as the neural basis of motor functions and movement-related disorders, spatial navigation and memory in the real world, spontaneous emotional affect and expression or social interaction.

To address these important questions, it is imperative for cognitive neuroscientists to perform neuroimaging studies in humans while they move naturally and behave in the real world. In turn, this requires the development of methods to record brain activity in such situations and an adaptation of contemporary experimental designs and analysis methods to account for the full complexity of real-world environments and rich human behavior. Recent years have brought a series of technological advances in mobile human neuroimaging, enabling brain recordings in freely moving humans and opening up a window into largely unexplored areas of cognitive neuroscience. In this Perspective, we highlight these new technologies, the novel findings they have recently enabled and their potential to provide insight into human cognition with unprecedented ecological validity. We discuss the limitations and challenges associated with mobile human neuroimaging methods and with the complexity of doing experiments in real-world environments. We argue that overcoming these challenges and using the synergy between research and development in this new area of mobile cognition will transform human cognitive neuroscience in the years to come.

Neuroimaging methods in humans

Traditional neuroimaging methods

Neuroimaging technologies — such as functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), scalp electroencephalography (EEG), magnetoencephalography (MEG) and intracranial electroencephalography (iEEG) — have given us invaluable insight into the neural underpinnings of human cognition¹. Before the recent technological developments that enabled mobile versions of some of these technologies to be created, human neuroimaging studies were limited in their ability to record brain activity during natural movement and behavior. Instead, studies typically recorded brain activity while participants viewed experimental stimuli presented on a screen in front of them, allowing participants to remain immobile. Furthermore, some of these methods (such as scalp EEG and fNIRS) are restricted to recording from superficial brain regions, making it difficult to analyze activity in subcortical regions such as the basal ganglia or medial temporal lobe (MTL), which are known to play central roles in a broad range of cognitive and behavioral functions.

Some traditional neuroimaging studies have indirectly studied the neural correlates of physical movement by measuring brain activity during imagined movement or the observation of another’s movement^19,20. Findings from imagined-movement studies have been particularly useful for building brain-machine interfaces (BMIs)²¹ using neural recordings in patients with paralysis^22,23. Despite theoretical models proposing that motor imagery and observation of movement engage brain mechanisms that are comparable to those involved in actual movement^24–27, only studies that facilitate direct comparisons with physical movements can establish the degree of this convergence.

Other studies have used virtual reality (VR; computer-generated models of the real world) to overcome some of the limitations of traditional neuroimaging. This has proven useful for studies that require immobility, since it can simulate mobile real-world experiences while the user is stationary and also allows the researcher control over stimuli presentation and environmental complexity²⁸. These studies have primarily used view-based VR, in which environments or avatars (virtual agents) are shown on a 2-dimensional screen and tasks can be performed using devices such as a keyboard or joystick²⁹. Other studies have used VR headsets, in which a wearable head-mounted display directly translates real-world movements, such as head rotations or physical walking behavior, to movements in the VR environment, and thus provides a more immersive experience²⁹. Studies using such VR headsets have allowed for physical movement, for example on an omnidirectional treadmill, to simulate navigation of real-world environments³⁰. Critically, however, these more immersive VR tasks are typically separated from neuroimaging sessions, in which participants are required to remain motionless. Consequently, it is unclear to what extent data from VR studies are reflective of real-life experience, in which brain activity is not only influenced by visual information but also by idiothetic self-motion cues (body-based sensory information arising, for example, from proprioceptive, vestibular and motor systems), rich environmental features and other behavioral or cognitive demands that may not be adequately captured in VR^12,13 (Fig. 1).

Fig. 1|. Experimental environments and factors impacting human cognition and behavior.

Examples of experimental environments used in cognitive neuroscience studies, including a motionless person in an fMRI scanner, performing a view-based virtual reality (VR) task shown on a screen (a); a person sitting in an experimental room, performing a view-based VR task shown on a screen (b); a person walking on an omnidirectional treadmill, wearing an immersive head-mounted display that allows physical movement within a VR environment (c); and a freely-moving person in a real-world environment (d). Examples of factors that can impact human cognition and behavior are shown at the bottom, including cognitive state (ongoing cognitive processes that that depend on task demands, attention, motivation or memory, for example); visual and auditory cues; idiothetic self-motion information (including proprioceptive, vestibular and motor cues); environmental ecological validity (the degree to which an environment reflects real-world settings and allows immersion in the scenario); and naturalness of movement and behavior (the degree to which it can reflect natural behavior in the real world). The degree to which each factor can be integrated and manipulated in each of the experimental environments is indicated. Visual and auditory cues, as well as participants’ cognitive state, can be experimentally manipulated in all example environments. However, idiothetic self-motion information cannot be involved in the scenarios shown in a and b and environmental ecological validity and the naturalness of movement and behavior are typically low in these environments. In the environment shown in c, idiothetic self-motion cues can be perceived, whereas environmental ecological validity and the naturalness of movement and behavior are typically moderate. An experiment in a real-world setting (d) can involve all factors, including idiothetic self-motion cues, high environmental ecological validity and natural movement and behavior.

Mobile neuroimaging methods

Technological advances have enabled the development of miniaturized and wearable scalp EEG equipment that allows mobile recordings of electrophysiological brain activity³¹. Recent studies have demonstrated that advanced analysis techniques can remove a reasonable amount of the motion artifacts to which scalp EEG signals are highly susceptible^32–36, making mobile scalp EEG a promising method for studying human brain activity during naturalistic movement and behavior (Fig. 2a). Moreover, while studies using scalp EEG have traditionally focused on the contributions of superficial cortical regions, recent studies have applied new source-localization methods for high-density scalp EEG recordings to analyze signals from deeper brain regions, such as the thalamus and retrosplenial cortex^37–40.

Fig. 2|. Recent advances in mobile neuroimaging methods.

a, Scalp electroencephalography (EEG) can record activation from superficial brain areas during natural movement and behavior. Recent methodological advances enable removal of movement-related signal artifacts and source-localization of scalp EEG signals in deep brain regions^32–40. b, Magnetoencephalography (MEG) traditionally requires participants to remain motionless within a restrictive scanner (left). However, recent technological advances allow recordings in moving participants via a mobile MEG system based on optically pumped magnetometers (OPM) that can be worn like a helmet (right)^48,49 c, Technical and methodological developments, such as the mobile deep brain recording and stimulation (Mo-DBRS) platform⁵⁸, enable recordings of brain activity in deep brain regions in freely-moving humans via chronically implanted closed-loop deep brain stimulation (DBS) devices. As part of the Mo-DBRS platform, a malleable metal arm holds a wand in place above the participant’s implanted device and is connected to a metal-framed backpack, allowing a real-time readout of intracranial electroencephalography (iEEG) activity. A post-operative X-ray image (top right) shows an example of a closed-loop DBS system implanted for the treatment of epilepsy. Magnetic resonance images (bottom right) show the locations of the recording contacts (red dots) and a schematic illustration of an electrode implanted in the medial temporal lobe.. Part c is adapted, with permission, from ref⁷¹.

The development of mobile fNIRS systems, combined with advanced motion-artifact correction methods, have opened up new opportunities to study the hemodynamic involvement of cortical regions during real-world experiences^41–47. Recent technological developments have further brought about a new generation of mobile MEG technology, based on moveable optically-pumped magnetometers (OPM-MEG)^48,49. Although wearable OPM-MEG systems are still dependent on ‘magnetic shielding’ (experimental indoor environments specifically designed to remove background magnetic fields), they are nevertheless another emerging method to non-invasively record brain activity from cortical and sub-cortical regions in moving participants (Fig. 2b).

Over the past decade, ‘closed-loop’ deep brain stimulation (DBS) devices have emerged, which continuously monitor iEEG activity through permanently implanted electrodes that can remain within a person’s brain indefinitely and deliver electrical stimulation upon detection of abnormal activity patterns⁵⁰. These systems have been implanted in thousands of individuals, primarily to treat epilepsy⁵¹ or Parkinson disease⁵² and, more recently, a wider range of neuropsychiatric disorders (including major depressive disorder (MDD])^53,54, post-traumatic stress disorder (PTSD)⁵⁵, obsessive compulsive disorder (OCD)⁵⁶ and binge eating disorder (BED)⁵⁷). As these devices are not externally visible and do not pose any obvious restrictions upon movement, individuals with such chronic implants can enjoy everyday activities. Thus, they provide a unique opportunity to obtain motion-artifact-free electrophysiological recordings from deep brain regions (such as the hippocampus, entorhinal cortex, amygdala and nucleus accumbens) in humans during natural movement and behavior^58,59 (Fig. 2c). Another advantage of these devices is that iEEG recordings can be obtained over long time periods (months or years), making data collection for longitudinal studies considerably easier than it is in traditional studies in which data is obtained during short-term recordings (over minutes or hours) in laboratory settings. Moreover, the stimulation capabilities of chronically implanted neural devices offer another window into human cognition: stimulation, triggered under specific conditions defined by the experimenter, allows the investigation of causal relationships between activation in specific brain areas and changes in behavior and cognition.

Although researchers have just begun to explore the opportunities that these new methods introduce, both non-invasive and invasive mobile recordings have already led to insightful discoveries in several sub-disciplines of cognitive neuroscience and promise, in future, to provide an unparalleled window into the neural mechanisms of naturalistic human cognition and behavior in the real world.

Potential of mobile human neuroimaging

The ability to conduct neuroimaging studies with freely moving human participants in real-world environments has the potential to provide key insights into the neural mechanisms of human cognition, both from a basic-science perspective (that is, with the general goal of understanding brain function and behavior) and from a clinical perspective (with respect to investigating malfunctions in neural systems and neurological or psychiatric disorders). Four areas of neuroscientific interest in particular serve as illustrative examples of the value of mobile neuroimaging methods: spatial navigation and memory, social cognition and interaction, movement and motor-related functions, and emotional affect and expression.

Spatial navigation and memory

Fundamental human abilities, such as navigating an environment without getting lost or forming memories of a personal experience, require the continuous processing and encoding of spatial information about one’s location, movements and the surrounding environment^60–62. Since deficits in navigational and memory functions are a hallmark symptom of memory-related disorders, such as Alzheimer disease (AD)^63,64, understanding how the human brain processes and encodes spatial information is also of critical importance for the development of therapies. Although spatial navigation and memory depend on a distributed network of brain regions, key components of this network are thought to be located in MTL regions, such as the hippocampus and entorhinal cortex^65,66. Recording MTL activity in moving humans, however, has not been possible until recently. Consequently, many questions remain as to whether and to what extent our current knowledge — derived from brain recordings in stationary participants — is reflective of physical navigation and memory processes in the real world.

A large body of research in rodents has demonstrated that navigation through the environment is accompanied by ongoing oscillatory MTL activity in the theta frequency band (~6–8 Hz). By contrast, VR navigation studies in stationary individuals with implanted electrodes indicated that, in the human MTL, these theta oscillations occur at lower frequencies (~1–4 Hz)⁶⁷. Recently, mobile intracranial recordings in humans were able to shed light on this discrepancy: it was found that theta oscillations at higher frequencies do exist in the human MTL when individuals physically navigate through their environment^16,68 and that the previously observed lower theta frequency activity may have resulted from the lack of physical movement in VR studies¹⁶. Moreover, mobile iEEG studies have demonstrated that movement-related theta oscillations in humans occur in bouts (which contrasts with findings in rodents, but is consistent with findings in non-human primates and bats^14,69,70) and are more prevalent during fast walking than during slow walking or stationary periods^16,68,71, revealing inter-species differences in the neural encoding of self-motion speed and the function of theta oscillations more generally. Another mobile human iEEG study demonstrated that the power of the theta oscillations is modulated by one’s location relative to environmental boundaries (such as the walls of a room)⁷¹ (Fig. 3a–b). Interestingly, theta power also reflected another individual’s location in the environment, suggesting that the human MTL uses common mechanisms to encode the location of oneself and others. It was further found that boundary-related theta power increases were only present when momentary location was behaviorally relevant (for example, when it contributed to success in a spatial memory task), suggesting that MTL representations of space can switch depending on cognitive demands. Together, these mobile iEEG studies demonstrated the impact of physical movement and the presence of others on neural activity during navigation and also provided novel insights into human brain mechanisms that would not have been possible without mobile neuroimaging techniques.

Fig. 3|. Studying spatial navigation and social interaction with mobile neuroimaging.

a, Schematic illustration of a mobile intracranial electroencephalographic (iEEG) study⁷¹ in which participants searched for and learned hidden target locations (yellow circle) or walked to visual cues (green rectangle). Black and red arrows signify examples of walking trajectories that pass close to environmental boundaries (walls) and inner room areas, respectively. Participants alternated between walking (walker) and observing (watcher). b, Summary of the results from the study⁷¹ illustrated in a, showing the walker’s medial temporal lobe (MTL) theta power increased when they were near a wall. Similarly, the watcher showed higher MTL theta power when the walker was near a wall. These effects only occurred when encoding of one’s own or the walker’s location was required (that is, to keep track of one’s own or the walker’s location before reaching a hidden target) c, An example of a potential future mobile iEEG study to investigate interbrain synchrony between two people at a cafe. The schematic traces above each image show hypothesized correlated brain activity between two individuals that increases during social (left) compared to non-social behavior (right). Parts a and b are adapted, with permission, from ref¹⁷⁵.

Mobile scalp EEG and fNIRS studies have also enabled novel insights into the neural mechanisms underlying spatial navigation and orientation during active movement^{39–41,72–75}. These studies found, for example, associations of cortical low-frequency oscillations with the speed and direction of physical movement^39,74,75 and suggested that oscillatory activity reflects multisensory information processing during physical navigation, evoked by a combination of visual, kinesthetic, vestibular and proprioceptive motion cues^72,73. Furthermore, the studies showed that decision-making (for example, route planning) periods during active navigation were accompanied by bursts of frontal-midline low-frequency oscillations⁷² and that interaction with the environment (touching walls, for example) affected spatial learning and the underlying oscillatory signatures in source-localized deep brain regions (such as the retrosplenial complex)⁴⁰. Studies have also used mobile scalp EEG to investigate how memories are formed during active movement in the real world. This revealed neural signatures of successful real-world item-context binding (that is, learning the location of an object in the environment) and demonstrated that the specifics of an environment (indoor versus outdoor, or the presence of prominent landmarks nearby) have a substantial impact on the successful formation of memories during real-world experiences ^76–78, providing another set of findings that were made possible only through mobile brain recordings done in natural environments.

While these studies showcase the potential of brain recordings in natural real-world scenarios, they reflect only the first steps towards understanding how the world — including one’s own movement or that of other individuals, objects, and environmental features — are represented in the human brain. Future mobile human neuroimaging studies are well positioned to account for the full complexity and richness of natural environments and human behavior to address questions that cannot be answered in immobile participants, such as how signals from different sensory systems are integrated and combined to form a coherent neural representation of space or a memory of an individual experience. They may also provide insight into how information from different systems is weighted in real-life situations in which some sources are imprecise and noisy (for example, vision in darkness). Recording brain activity in freely moving humans also enables the study of the individual contributions of different sensory cues to spatial representations. For example, studies could compare the impact of different sensory input modalities on theta oscillations during natural physical movement and the effect of conditions in which sensory information is limited (such as moving in a wheelchair or in darkness, or when movement is simulated in VR). Moreover, while previous work on human navigation primarily used environments with a flat surface, mobile neuroimaging methods will allow for the investigation of the encoding of volumetric space and movements in the vertical dimension (such as walking up or down staircases or climbing) in the human brain. Furthermore, mobile methods can provide insights into how the human brain encodes spatial information during dynamic scenarios that require interaction with one’s environment or with other individuals that are moving through space (such as how a soccer player’s brain keeps track of multiple moving players). Another important future area of study will be the interplay between the neural representations of space and other cognitive processes, such as episodic memory formation (which involves the integration of information about space, time, sensory cues, emotional states and other contextual information) or decision-making (such as during route planning) in complex real-world situations that involve movement.

Utilizing the DBS capabilities of implanted devices, future studies will also be able to investigate whether stimulation of subregions within the MTL has an impact on navigational function or the formation of episodic memories during real-life experiences. Such causal tests of brain–behavior relationships have previously been possible only under restricted and non-natural conditions in stationary participants (using invasive^79–81 or non-invasive stimulation⁸²), but can now be performed over the short- or long-term during real-world experiences. Moreover, understanding the neural mechanisms underlying navigational and memory functions will provide unique insight into the causes and possible interventions for brain disorders that affect these functions, such as AD, epilepsy or PTSD. For example, it has been shown that changes in navigation- and memory-related MTL activity are among the earliest signs of AD^63,83. Since thousands of epilepsy patients have chronically implanted recording electrodes in MTL regions, mobile long-term recordings from these patients could be used to identify AD-related biomarkers in longitudinal studies and enable efficient methods for early detection of the disease. This would, in turn, allow administration of treatment options at the earliest disease stages, together with long-term iEEG recordings for measuring long-term treatment response in affected brain regions.

Social cognition and interaction

A critical aspect of human cognition is participation in societal structures through complex social interactions. Social neuroscience seeks to determine how social interactions dynamically modify cognition, behavior and the brain activity of interacting individuals^84,85. Decades of social psychological research have highlighted personal and environmental factors that modulate social interactions and relationships between individuals. Critically, this has included investigation of relationships in multi-individual scenarios such as dyads and groups ^86–89. This work has led to the development of conceptual frameworks and theoretical models of social behavior and cognition — including the dual-process model^90,91 (in which both implicit and explicit processes can exert influence on social behavior and/ or cognition), theory of mind⁹² (which concerns the ability to infer another’s mental state) and social complexity theory ⁹³ (in which a group of interacting individuals is viewed as a complex adaptive system) — that have been applied to study the behaviors of human and non-human collectives. For example, concepts from complex systems theory such as criticality⁹⁴ (describing a situation in which individuals adaptively switch into and out of coordinated group behaviors) and emergence (the situation in which unique group behaviors arise that were non-existent at the individual level) have been used to investigate individual, dyadic, and collective behavior^95,96 (such as coordination and cooperation) in both humans^97–99 and non-human animals¹⁰⁰. Given that humans exist and behave as part of nested hierarchies of collectives within collectives, a more complete understanding of optimal and disordered individual and community function is needed. This will require multi-scale approach¹⁰¹ in which movement-robust forms of neuroimaging are applied in groups within naturalistic environments where emergent behaviors arise in response to manipulated environmental pressures and incentives.

Prior non-invasive immobile neuroimaging studies in humans have developed creative ways to study social behaviors in stationary participants through the presentation of images of others or social scenes, pre-recorded videos depicting social interactions, 2-person live video (dual-video) and VR avatars that are updated in real-time based on participant actions. These studies have investigated a myriad of social behaviors, including emotional face processing, joint gaze and attention and implicit and explicit bias^102,103. For example, dual-video studies (using fMRI or MEG) in dyads have highlighted how interactive behaviors (such as competitive games, facial emotional expression and recognition, and speaker–listener communication) modulate coordinated brain activation patterns (interbrain synchrony) across individuals^104–107. Another dual-video study using scalp EEG identified increased interbrain oscillatory synchrony when one person mimicked another person’s hand movements in real-time¹⁰⁸. However, while the findings of these and other studies have accelerated the emerging field of social cognitive neuroscience and elucidated several neural mechanisms underlying social behavior and cognition, they did not involve live face-to-face interaction, which has been shown to rely on distinct neural mechanisms and to amplify interbrain synchrony between individuals ^102,109,110.

Mobile non-invasive neuroimaging studies of social interactions in humans have allowed researchers to extend the findings from studies conducted in immobile individuals by recording brain activity simultaneously from groups of participants interacting face-to-face in laboratory or real-world environments: this is typically referred to as ‘hyperscanning’^102,103,111. Mobile scalp EEG hyperscanning studies have yielded several findings, including the identification of real-world behavioral interactions that modulate interbrain synchrony^112,113. For example, one multi-person study recorded from groups of high school students during several sessions over the course of a semester in the classroom and found that the degree of interbrain oscillatory synchrony across classmates predicted class engagement and social dynamics, highlighting the utility of longitudinal and multi-person studies in real-world settings¹¹². Another study recorded from multiple dyads of museum visitors and found that interbrain oscillatory synchrony was modulated by their levels of empathy, closeness, engagement, joint action and eye contact, emphasizing the importance of face-to-face experiences that can modulate neural mechanisms of social cognition¹¹³.

Recent developments in mobile invasive neuroimaging methods are poised to build on current understanding by investigating how deep brain structures are involved in spontaneous real-world human interactions during movement. Indeed, a key advantage of mobile invasive neuroimaging methods is the opportunity to study spontaneous social interactions in the real world: since iEEG recordings are continuous, it is possible to minimize study participants’ awareness that they are being observed and thus the resulting regulation of social behaviors, as compared to a laboratory-based experiment. Future mobile iEEG studies could, for example, examine neural dynamics related to the generation and perception of emergent affective states, natural facial expressions, vocalizations, gestures, involuntary body mimicry and other social behaviors. These could then be compared to neural dynamics during non-social behaviors (Fig. 3c). Other possible future investigations could explore how subcortical regions support the complex social dynamics that emerge in group settings (including compassion, empathy and altruism) and that may influence cognitive processes (such as decision-making and implicit or explicit bias), enable one to predict another’s future actions during cooperative or competitive interactions, or differentially encode social and non-social rewards. It will also be interesting to study how social relationships or status (for example, competitor versus cooperator, leader versus follower or in-group versus out-group) develop during short-term interactions or over repeated long-term interactions, and how these relationships modulate individual and interbrain dynamics. Furthermore, when combined with non-invasive mobile neuroimaging methods (using simultaneous iEEG and mobile scalp EEG, for example), there is an opportunity to investigate both subcortical and cortical activity during spontaneous, emergent group behaviors in naturalistic settings and thus to investigate interbrain network activity. Such mobile neuroimaging setups could also be used to perform interbrain analyses investigating the dynamic properties of individual brains and the extent to which they form part of a coherent collective-level computational entity. This will permit the study of multi-scale competency, which is the concept that effective behaviors can be observed across each level or scale of the social hierarchy, from the individual to the group and organization level. The results from these studies will provide deep multi-scale insights into complex social systems underlying the adaptive and maladaptive patterns that can occur in spontaneous human group behavior¹¹⁴, how individual brain activity gives rise to self-organizing, intelligent collective behaviors, and how collectively intelligent behaviors are shaped by the context in which they occur. Further, this multi-scale perspective may be necessary to fully understand psychopathologies and the broader social structures that individuals with psychopathologies are embedded within. Measurements of community function hinge on assessing how individuals operate within their community, as well as to examine collective behaviors at a larger scale.

Beyond the intrinsic importance of understanding basic neural substrates, numerous disorders affect social behaviors and community function^115–117, including autism spectrum disorder^118,119, attention deficit hyperactivity disorder¹²⁰, social anxiety disorder¹²¹, bipolar disorder¹²², schizophrenia¹²³, and borderline personality disorder¹²⁴. Mobile invasive and non-invasive neuroimaging approaches provide an opportunity to investigate these disorders in ecologically-valid settings. This could facilitate the identification of electrophysiological biomarkers of behavioral symptoms to assist with earlier detection of difficult-to-diagnose disorders and the development of more targeted and neurophysiologically informed interventions¹²⁵ as well as allowing us to evaluate how whole-body movements (such as volitional or involuntary body mimicry) may be altered. Thus, advancing our understanding of the neurophysiology of social interaction will build a foundation on which to improve our approaches to treating disorders that affect social behaviors.

Movement and motor-related functions

Humans engage in complex movement patterns, such as walking, gesturing and vocalization, which can co-occur and involve the rapid coordination of multiple muscles. Non-human animal studies have shown that neural activity patterns differ during free movement relative to constrained behaviors^126,127. How the human brain precisely controls volitional and complex movements remains unclear, in part owing to the motion constraints of traditional human neuroimaging approaches; however, such knowledge would serve a significant role in informing the development of interventions for movement-related disorders.

The ability to walk (that is, one’s gait) enables mobility and independence in daily life. The availability of mobile neuroimaging techniques (such as scalp EEG) has enabled studies on the neural correlates of gait in ambulating humans. This work has identified fluctuations in brain activity in motor and somatosensory regions in relation to the gait cycle^128–131, and these findings have been confirmed with mobile iEEG recordings from the motor cortex¹³⁰ and basal ganglia¹³¹. Specifically, these studies reported modulations in amplitude of oscillatory activity in the upper beta and high gamma frequency range that encoded movement onset, termination, muscle synergies and freezing of gait in individuals with Parkinson disease. Decoding such parameters from neural signals is an essential prerequisite for the implementation of neuroprosthetic devices, such as those intended to provide spinal cord stimulation for gait rehabilitation or restoration¹³². Another study investigated the cortical mechanisms underlying conscious modification of ongoing walking compared to natural walking and identified decreases in the power of multiple oscillations at particular phases of the gait cycle during gait modification¹³³. These findings revealed the nuanced neural dynamics that support intentional movement modification and provided key insights that could inform the development of BMIs for rehabilitation^133,134. Another critical aspect of natural walking in daily life is the ability to rapidly navigate around obstacles, a skill that can become impaired in advanced age and motor-related disorders^135,136. One mobile scalp EEG study showed that the updating of motor plans occurred at the time that an obstacle appeared, rather than when an individual arrived at the obstacle and that beta oscillations (~13–30 Hz) marked the traversal of an obstacle¹³⁷. Mobile neuroimaging approaches have also suggested overlapping neural mechanisms during real and imagined movement²⁴, which is consistent with BMI studies showing successful control of robotic arms in healthy individuals¹³⁸ and patients with paralysis^22,23. Mobile non-invasive neuroimaging approaches have also enabled the investigation of expressive movement, such as dance, within a burgeoning field of research looking into the neural computations underlying complex coordinated motor action, memory and timing¹³⁹.

Research on the neural correlates of cognitive–motor interference has also been possible in studies in which mobile neuroimaging techniques have been coupled to paradigms in which two tasks must be completed simultaneously (for example, performing a cognitive task while executing a secondary motor task). Both cognitive performance and motor behavior have been shown to depend on whether a task is performed in an ambulatory or stationary format^32,140,141. More specifically, the neural responses observed in classical cognitive paradigms (such as Go/NoGo tasks or oddball tasks) appear to be altered when they are attempted during walking compared to immobility, highlighting the importance of investigating cognitive–motor interference in real-world and ambulatory scenarios^32,140,142. Another study focusing on dual motor tasks showed that sensorimotor rhythms are differentially modulated in young and elderly (>70 years) participants, suggesting that age-related decline may further affect the competition for cognitive resources¹⁴¹. Another approach used to investigate cognitive–motor inference involves cueing a motor behavior modification in response to visual or auditory cues or asking participants to maintain motor behavior in the presence of physical or visual distractors^143,144. In the former scenario, cognitive processes must translate sensory stimuli into motor behavioral modification, while in the latter, the visual or physical perturbations elicit corrective motor movements (for example, the act of balancing to prevent falling¹⁴³).

While these approaches have provided valuable insights into the cortical neurophysiology of natural and spontaneous movement, recent advances in clinical treatment of movement disorders have provided an opportunity to chronically record longitudinal iEEG from deeper brain structures in individuals moving around during everyday activities. Traditional pharmacological approaches to treatment of movement disorders such as Parkinson disease and dystonia often fall short and thus chronically implanted DBS devices within areas such as the subthalamic nucleus, ventral anterior or lateral thalamus and globus pallidus have become a standard treatment⁵². Recent technological advances have enabled closed-loop stimulation and long-term chronic recordings from these devices for research studies and in clinical trials^145–148. These approaches enable stimulation to be more specific: for example, locked to a particular oscillatory phase or a specific behavioral metric such as a particular movement rhythm. Behavioral measurements can be captured, using accelerometers or electromyographs that are integrated into wearable sensors such as smart watches, and used to ensure that stimulation coincides with specific phases of motor behavior, such as a patient’s tremor rhythm¹⁴⁹. Thus, a new subfield of research has emerged in which deep brain mechanisms of movement-related functions and disorders can be studied outside of the traditional laboratory or clinic during real-world behaviors and daily activities, and in a longitudinal setting.

Longitudinal studies using these mobile iEEG methods in individuals with movement disorders have been used to identify the neurophysiological mechanisms associated with symptom fluctuations and aberrant movements. These studies have highlighted, for example, the pathophysiological role of cortico-subthalamic circuit theta oscillations in cervical dystonia dyskinesias and subthalamic beta activity in rigid or akinetic states in Parkinson disease^145,147. The chronic and mobile aspects of these brain recordings, when combined with on-body wearable sensors, provide a means to study the mechanisms underlying successful or unsuccessful movements. Furthermore, they could enable the observation of the neurophysiological mechanisms that relate to spontaneous motor behaviors associated with daily activities — such as walking, reaching, grasping, driving or speaking — or behaviors that pose challenges for patients^148,150. Moreover, such real-world brain-behavior quantification and synchronization studies create an opportunity to identify clinically meaningful neurophysiological signatures that can, in turn, improve closed-loop treatments (Fig. 4a). For example, one mobile iEEG study, in which 2,600 hours of at-home recordings were collected, found that increased oscillatory coherence between the subthalamic nucleus and motor cortex at particular frequencies differentiated between dyskinetic on and off states in a real-world setting¹⁴⁵. These results demonstrate how long-term mobile iEEG can be performed with minimum intrusion to normal activities and how such research studies can provide insight into spontaneous symptoms without the need to artificially elicit motor deficits in a laboratory setting. Similar studies in the future could allow for personalized treatment approaches and are also poised to illuminate the neurophysiological mechanisms underlying successful movement patterns.

Fig. 4|. Studying motor-related functions and emotional affect with mobile neuroimaging.

a, There are opportunities to develop personalized treatments and advance our understanding of the neural mechanisms underlying motor-related functions (top) and psychiatric symptoms (bottom) in individuals with a range of conditions using chronic mobile intracranial electroencephalographic (iEEG) recordings. The schematic shows example symptoms of two disorders that might be targeted with this approach — Parkinson disease and major depressive disorder — and illustrates how closed-loop deep brain stimulation (DBS) might be tailored to relieve symptom burden. Insights gained through continuous iEEG recordings in patients after symptom amelioration can advance our knowledge of normal motor-related functions and affective behaviors (right), which in turn enables progress in designing effective treatment approaches for affected patients. b, An example of a study in which long-term (~30hr) mobile iEEG recordings were obtained from electrodes implanted in the left ventral capsule/ventral striatum (VC/VS) and right bed nucleus of the stria terminalis (BNST). The timing of the collection of iEEG data, biometric heart rate monitoring and behavioral obsessive compulsive disorder (OCD) intensity ratings are shown at the top for one participant ⁵⁶, together with a 90-second segment of the data shown below. c, Self-reported OCD symptom intensity (on a scale of 0–10) negatively correlated with normalized delta power (0–4 Hz) recorded in the VC/VS 1-minute before and after each self-report from the patient shown in (b), captured across 3 days of continuous recording. The black line represents the least-squares fit. Parts b and c are adapted, with permission, from ref⁵⁶.

As more individuals receive treatment via closed-loop DBS devices, not only for Parkinson disease but also epilepsy and other disorders, there will be increased data available to help us understand inter-individual differences and similarities in the neural mechanisms underlying normative and pathological motor-related (dys)functions. Together with non-invasive mobile neuroimaging methods, this will give an opportunity to study simple and more complex coordinated movements (such as those seen during exercise, sports or dancing), to illustrate how these movements are learned, initiated, modified, and to develop personalized treatment approaches for movement-related disorders.

Emotional affect and expression

Research in the field of affective neuroscience (the study of how the brain processes emotion) has led to the prominent view that affective processes are so tightly integrated at cognitive and neural levels that it is impossible to separate them^151,152. Many cognitive processes (including memory, attention, language and decision making) are strongly influenced by and interact with emotional states to contribute to behavior¹⁵². Moreover, a large body of research has demonstrated that emotions not only evoke bodily feedback (such as movements in response to an aversive stimulus) but that bodily experience and emotions influence each other in a bi-directional fashion¹⁵³. Due to the strong interplay between affective processes, bodily experience and cognition, it has been suggested that emotions should be modeled holistically, as whole brain-body phenomena¹⁵⁴.

Studying emotional processes while taking these brain-body dynamics into account poses a major challenge for affective neuroscience. Non-mobile neuroimaging methods limit or eliminate the impact of bodily sensations (including body movements, gestures or facial-muscle contractions), thereby neglecting these critical interactive dynamics. In addition, experimental access to emotional states is challenging because of the difficulty in eliciting naturalistic behaviors in laboratory settings. While laboratory-based studies of emotion primarily use behavioral paradigms that involve the presentation of images, film clips and cues for autobiographical recall, it is unclear to what extent these techniques elicit emotional responses with the same strength and of the same quality as natural emotionally-laden experiences in the real world¹⁵⁵. Finally, emotional states can evolve and fluctuate over long time courses (days, months and years), making them inherently difficult to study in laboratory experiments.

Mobile neuroimaging methods provide opportunities to overcome these limitations by enabling neural recordings during emotional experiences in real-world situations, without restricting the individual’s bodily response. For example, mobile iEEG studies in patients with chronic neural implants can record from emotionally-relevant brain regions (such as the nucleus accumbens, amygdala and prefrontal cortex¹⁵²) during significant events (such as the birth of a child, death of a loved one, birthdays or weddings) or in fearful situations (such as scary rides, haunted houses or traumatic events). This can thereby provide unique insights into the neural mechanisms underlying naturalistic fluctuations in positive or negative affect or during the formation and expression of fear-related memories and their interaction with associated bodily responses. Moreover, long-term recordings can determine how neural activity changes over long timescales not only at the time of the event but also in the days, weeks and years that follow.

Another promising area of investigation for mobile brain recordings is the impact of environmental settings and physical activity on emotion and mental health¹⁵⁶. For example, mobile scalp EEG studies have investigated the effects of different real-world settings (including busy urban versus quiet green spaces) and activities (such as active exploration versus passive viewing) on brain activation and associated emotions^157–160, providing insight into mechanisms of emotional and mental recovery from stress and fatigue that would not be possible in stationary and laboratory-based studies. Moreover, mobile non-invasive neuroimaging has been used increasingly for emotion recognition, in which neural signals are automatically classified in real-time to determine an individual’s emotional or mental state¹⁶¹. The future potential of non-invasive emotion-recognition systems for both research and industry seems endless, ranging from areas of human-computer interaction and artificial intelligence (for example, to adapt computer, robot or game behavior with respect to user experience), e-learning (for example, to adapt online course material to a student’s mental state), entertainment (for example, music players that select content based on the user’s mood), to medical technology (such as remote and long-term tracking of treatment progress in individuals with MDD).

Psychiatric disorders are widespread and, despite pharmacological and psychotherapeutic approaches, a substantial population of patients remain resistant to treatment¹⁶². Ongoing efforts utilizing closed-loop DBS devices to treat mood and anxiety-related disorders such as MDD^53,54,163, PTSD⁵⁵ and OCD⁵⁶ have yielded promising results and are ushering in a new era of research on emotional affect and expression. Closed-loop DBS devices in these patients enable mobile iEEG recording that can be combined with accessible wearable biometric sensors and symptom-tracking devices. As an example, one study⁵⁶ combined over 1,000 hours of at-home mobile iEEG data, heart rate measurements, and OCD symptom intensity ratings to investigate the neurophysiological changes in deep brain regions associated with fluctuating and spontaneous anxiety-related symptoms (Fig. 4b–c). This revealed a negative correlation between ventral capsule (VC) and ventral striatum (VS) delta power and OCD symptom intensity and showed that VS stimulation was associated with increased positive affect. A separate pair of studies in an individual with treatment-resistant MDD^53,54 found symptom-specific neurophysiological activity that responded to VC/VS stimulation in a dose-dependent manner, and led to clinical improvement when used to trigger closed-loop DBS. Together, these studies highlight a key advantage of chronic mobile invasive neuroimaging methods: they allow for the assessment of spontaneous and natural symptom fluctuations over long time periods, which would not be possible in short-term laboratory-based studies. Such mobile long-term recordings may further enable the identification of novel disease-related biomarkers and the development of personalized closed-loop DBS treatments that are more likely to be effective in real-world settings (Fig. 4a). Additionally, short- and long-term tracking of treatment responses will provide insight into both the disease state and the typical physiology (during asymptomatic periods) of human emotion in the real-world. This is important because human emotions are highly influenced by small and large factors of internal state, relationships with surrounding individuals and day-to-day experiences, all of which are difficult to simulate and control in laboratory settings.

In sum, mobile neuroimaging studies of emotional affect and expression, combined with wearable technologies can allowing researchers to observe the development and full cycle of authentic emotional states as they occur spontaneously within the context of an individual’s day-to-day life and over natural timescales.

Challenges of mobile human neuroimaging

Technological advances that enable the measurement of brain activity during naturalistic behavior and in dynamic real-world environments also bring along new challenges for experimental design, data acquisition and analysis, as well as the interpretation of findings. Traditional laboratory studies are often designed to maximize experimental control by isolating a few variables of interest (considered the ‘signal’) while limiting or excluding the impact of irrelevant variables (considered ‘confounds’ or ‘noise’)^7,164. By contrast, real-world settings contain many variables that are considerably harder to control. For example, in natural social settings, a research participant’s behavior and cognitive processes might be influenced by luminance, sounds, odors, distracting environmental elements, one’s own and other people’s movements, complex emotional and interpersonal factors and the behaviors of others. Conducting such studies thus requires innovative approaches for experimental design to provide an adequate degree of experimental control (by eliminating or reducing unwanted influences) while at the same time allowing an appropriate and analytically manageable level of naturalism and uncontrolled influences to enable ecological generalizability. One step towards studying completely natural behavior in the real world is to design studies that include variables that have previously been considered noise or confounds into the experimental design and analysis strategy. This, in turn, requires data acquisition and synchronization solutions that enable the recording of multi-modal data, so that all variables (those of primary interest as well as possible confounds) can be analyzed post-hoc. Rapid recent progress in the development of wearable technologies has enabled simultaneous recording of multiple data streams, for example using body-motion and eye-tracking systems, physiological sensors and audiovisual recordings (Box 1).

Box 1: Multi-dimensional recordings for mobile behavioral studies in humans.

Studying freely moving participants in complex (that is, real-world) scenarios requires multi-dimensional recordings of numerous variables, including those considered to be ‘noise’ or ‘confounds’, in order to be able to include these variables in the study design and analyze their impact on human cognition and behavior. Besides mobile neuroimaging data, recording such multi-dimensional data is possible through the use of modern technological wearable measurement systems that can record behavioral, physiological and environmental influences (see the figure). For example, within indoor environments, the movement of people and objects can be recorded with optical motion tracking systems. These systems continuously capture the position of reflective wearable markers with sub-millimeter resolution through motion tracking cameras and this can be used to analyze positional or directional data (such as one’s location in the environment or movement speed) or even biomechanical information (such as gait analysis). Alternatively, an inertial measurement unit (IMU) system, using a combination of accelerometers, gyroscopes and magnetometers, can be mounted on multiple parts of the body to track motion without the need for stationary equipment such as cameras. IMUs can thus be used to capture movement and positional or orientation information more flexibly in completely natural indoor or outdoor environments, although they provide lower spatial precision than optical motion tracking systems. Audio and video recordings (captured with body-mounted cameras, for example) enable recordings of environmental cues. Mobile eye-tracking systems allow measurement of eye movements, pupil position, pupil size and gaze position and this can be mapped onto video recordings of the participant’s view in order to determine their focus of visual attention at any point in time. Wearable systems also enable the measurement of physiological parameters such as heart rate, respiration or skin conductance. These recording systems can be combined with various stimulus presentation technologies, such as virtual reality (VR) headsets (head-mounted displays), augmented reality (AR) headsets (which allow the integration of digital elements, such as virtual objects, into the real-world environment) or used without digital stimulus presentation technology in real-world studies. Outside the laboratory, wearable devices such as smart watches and other portable handheld devices can be used, often in combination with specifically-designed smartphone apps that can capture additional variables such as physiological measurements (including heart rate or acceleration), symptom ratings, or sleep quality^2,5,6,56. Importantly, the data streams from multiple recording systems need to be synchronized (temporally aligned) with each other and with data from neural recordings to allow a precise analysis of an individual variable’s influence. This synchronization can be achieved via ‘marker’ signals sent to the individual recording system (if they allow input of such signals) or via separate technical solutions (such as the recording of ‘timestamps’ for individual events, optical or auditory marker signals captured by audiovisual recordings or network time protocols)¹⁴⁵. Most of the required technical solutions (computer code, for example) for data synchronization have been implemented and are openly available for the Mo-DBRS platform⁵⁸ (a technical platform to enable deep brain recordings and stimulation through implanted neurostimulation systems) or from the Open Mind Consortium (a consortium formed to accelerate cooperation and innovation in the use of implantable neurostimulation systems).

Given the large number of data streams that can be collected, data analysis methods must appropriately accommodate for the added complexity. There is a need for advanced mathematical methods and models (including mixed-effect models, multimodal models and multivariate methods), similar to those that have been used for behavioral studies in freely moving rodents and primates (including humans)^{9,14,71,165–167}. Furthermore, it will be imperative to draw techniques from computational ethology, a field that employs automated methods (including machine learning, deep neural networks and computer vision) to quantify behavioral motifs and rich stimuli in the natural world and complex virtual environments³. The use of these advanced analytical approaches will help to capture and explain the ‘noisiness’ (that is, the variability in the ways that different individuals express a particular behavioral motif) of naturalistic behaviors. For example, computational ethology methods could be used to quantify behavioral variables such as cooperativeness based on body language and facial expressions¹⁶⁸. These methods could also be applied to complex environments (for example, by automating classification of types of landmarks) to extract and interpret meaningful conclusions from humans’ interaction with others as well as their environmental setting³, as demonstrated in previous studies^169,170. These advanced methods will help link multi-dimensional measurements of behavior to cognitive states and neural activity.

One limitation of mobile iEEG recordings is the selective coverage of electrode implantation sites, which are determined by a given individual’s clinical indications⁷⁹. There is always a limited number of simultaneous recording sites (usually 4) per person. This limitation can be offset to some extent by recruiting many participants for these studies (which may become more feasible given the large and growing population of these patients) to allow for data collection across many pooled electrode channels. Furthermore, recent technological developments allow mobile iEEG recordings (and even single-neuron activity) in individuals with a larger number of electrodes implanted (>100 recording channels), for example those who are undergoing epilepsy monitoring in hospital settings¹⁷¹. The range of brain structures that can be recorded from is also selected based on clinical need. Currently, the most common regions implanted are in the MTL and basal ganglia¹⁴⁵ and also, but less frequently, the frontal, occipital, insular, and parietal areas⁷⁹. In addition, investigational device exemptions (IDE) have been acquired to utilize these devices for treatment of cervical dystonia¹⁴⁷, MDD⁵⁴, PTSD⁵⁵, BED⁵⁷ and OCD⁵⁶, which has expanded the range of regions implanted to include areas such as the subthalamic nucleus, motor cortex, VC and nucleus accumbens. Non-invasive mobile neuroimaging approaches (such as mobile scalp EEG, OPM-MEG, and fNIRS) can circumvent these limitations by offering broad coverage of brain regions, although the signal is most robust from superficial structures. Non-invasive recordings could also be combined with iEEG recordings to offer a dual approach that combines recordings from deep and superficial structures for unprecedented investigations.

Another challenge arises from the fact that mobile iEEG recordings can be obtained only from people with medical conditions (such as epilepsy^51,172,173), limiting the extent to which the results can be generalized to healthy participants. Hence, several strategies should be adopted when conducting these studies, to minimize the impact on the interpretability and generalizability of the data obtained. The analysis of data from individuals with particular conditions requires specialized algorithms to detect and exclude time periods of abnormal (epileptic, for example) brain activity in recordings¹⁷⁴. Furthermore, since recordings occur over the long term, researchers can view historical data and recruit participants with a low number of abnormal events, including those that have shown positive clinical response to treatment. With these approaches, previous studies have found that ~3% of the data during experimental sessions was affected by epileptic activity^58,68,71, leaving most of the data available for testing hypotheses related to intended research questions. Moreover, differences in neural dynamics that are observed across two or more behavioral conditions (within an experimental task) and/or clinical conditions (such as epilepsy versus Parkinson disease) would help minimize the possibility that brain-behavior relationships are disease-related. With the rise in implantable neural devices being used across a wider range of neuropsychiatric disorders there may be an increased generalizability of findings independent of disease. Additionally, keeping records of participant’s medications and including them in statistical models should be considered. Non-invasive neuroimaging techniques enable recordings in healthy participants. Studying the human brain with (both stationary and mobile) non-invasive methods thus provides an excellent opportunity to carry out complementary and comparative studies, to contrast data between patients and healthy participants, and to identify disease-related influences on neural and behavioral findings. While these strategies are critical to enable meaningful interpretations of data, it should be noted that most people with epilepsy and other clinical conditions are able to function normally in society and to perform many cognitive functions during everyday experiences – thus, the underlying neural mechanisms should and can be studied.

Conclusions

The standard approach in contemporary cognitive neuroscience is to perform neuroimaging studies under strictly controlled conditions in laboratory settings, to isolate and investigate basic principles about neural mechanisms of cognition. It is often assumed that the findings will generalize from these controlled experimental settings to natural human behavior^7,164; however, to date it is largely unknown to what extent most of our knowledge reflects the complexity of cognitive processing under natural conditions. Until recently, many of the models and theories developed for or derived from laboratory experiments often could not be tested in real-world setting due to the limitations of traditional human neuroimaging methods. Now, enabled by innovative advancements in mobile neuroimaging techniques, wearable behavioral devices to record physiological and environmental influences, and advanced computational data analyses, neuroscientists are able to investigate complex cognitive processes in freely moving and naturally behaving people that are exploring and interacting with their environment. These new technologies have the potential to enable first-in-human discoveries and the development of novel treatments for neurological and psychiatric disorders that are more likely to translate to real-world settings³⁴.

We argue that for the cognitive neuroscience field going forward, real-world studies of human cognition with mobile neuroimaging methods will be critical, to test whether models that have been developed under laboratory conditions hold true in natural settings. Importantly, however, the value of mobile brain recordings goes far beyond post-hoc validation (or invalidation) of laboratory-derived models. Instead, insights and perhaps unexpected findings from real-world studies will provide a foundation from which new theories and models can be developed. Thus, rather than a unidirectional approach, starting from a theory to be tested in the laboratory and validated in the real world, it will be critical to follow an approach where theories, laboratory-based experiments, and real-world studies inform each other in a back-and-forth fashion, with the ultimate goal of gaining an ecologically valid understanding of human cognition and malfunctions in underlying neural systems. Lastly, further developments in mobile neuroimaging technologies, such as electrodes with the ability to record single-neuron activity or neurochemical signals during everyday life, will continue to speed up scientific progress in the years to come.