Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Jan 1.
Published in final edited form as: Int J Psychophysiol. 2013 Sep 26;91(1):22–29. doi: 10.1016/j.ijpsycho.2013.09.003

Imaging natural cognition in action☆

Klaus Gramann a,b,*, Daniel P Ferris c, Joseph Gwin c, Scott Makeig d
PMCID: PMC3983402  NIHMSID: NIHMS528018  PMID: 24076470

Abstract

The primary function of the human brain is arguably to optimize the results of our motor actions in an ever-changing environment. Our cognitive processes and supporting brain dynamics are inherently coupled both to our environment and to our physical structure and actions. To investigate human cognition in its most natural forms demands imaging of brain activity while participants perform naturally motivated actions and interactions within a full three-dimensional environment. Transient, distributed brain activity patterns supporting spontaneous motor actions, performed in pursuit of naturally motivated goals, may involve any or all parts of cortex and must be precisely timed at a speed faster than the speed of thought and action. Hemodynamic imaging methods give information about brain dynamics on a much slower scale, and established techniques for imaging brain dynamics in all modalities forbid participants from making natural extensive movements so as to avoid intractable movement-related artifacts. To overcome these limitations, we are developing mobile brain/body imaging (MoBI) approaches to study natural human cognition. By synchronizing lightweight, high-density electroencephalographic (EEG) recording with recordings of participant sensory experience, body and eye movements, and other physiological measures, we can apply advanced data analysis techniques to the recorded signal ensemble. This MoBI approach enables the study of human brain dynamics accompanying active human cognition in its most natural forms. Results from our studies have provided new insights into the brain dynamics supporting natural cognition and can extend theories of human cognition and its evolutionary function — to optimize the results of our behavior to meet ever-changing goals, challenges, and opportunities.

Keywords: Mobile brain imaging, EEG, ICA, Natural cognition

1. Introduction

More than a century of neuroscience research related to human cognition has revealed important insights into the architecture of the human cognitive system, its underlying anatomical structure, and supporting physiological processes. Using established brain-imaging modalities including positron emission tomography (PET), single photon emission spectroscopy (SPECT), and functional magnetic resonance imaging (fMRI), remarkable progress has been made in several areas. During the last decades important advances have occurred in understanding the functional architecture of the human visual system (Mishkin and Ungerleider, 1982; Goodale and Milner, 1992), attention (Corbetta and Shulman, 2002; Handy et al., 2003), the mirror neuron system (Iacoboni et al., 1999; Rizzolatti, 2005), and systems supporting human memory (Squire and Zola-Morgan, 1991; Squire and McKee, 1993; Gabrieli et al., 1997), emotion (Damasio, 1996), motor control (Paus et al., 1993), or the so-called default mode or rest network (DMN, Raichle et al., 2001). These and several other investigations have provided important insights forming the basis for advancing theoretical frameworks describing the complex architecture of human cognition and its underlying neural principles.

2. Restrictions of traditional brain imaging approaches

While these established brain-imaging analysis methods are still being improved and will remain important research tools, new techniques are now required for studying cognition under a more general range of conditions that include natural motor behavior. While the need to optimize the outcomes of motor behavior is arguably the principal driver of brain evolution, most current brain imaging experiments require the participant to hold their head in a fixed position during data acquisition to avoid serious signal artifacts contaminating the brain signal of interest. To ensure this, participants are allowed to make at best minimal movements during scanning, typically digital finger button responses conceptualized as point-like processes without duration or spatial extent.

However, much of our cognition is tightly coupled to our motor actions in an ever-changing environment and to evaluating behavioral outcomes on multiple scales. This coupling of cognition to action includes continuous active selection of information (rather than passive reception and interpretation of suddenly presented stimuli), continuous or intermittent active manipulation of the environment (rather than passive observation of environmental changes), near-continuous active integration of movement-related idiothetic information received from the body within one or more relevant external reference frames (rather than only sensory information from stimuli presented in a fixed reference frame, e.g., the face opposing plane of a computer monitor), and active prediction and evaluation of continuously evolving outcomes of current behavior including their environmental sequelae (rather than maintained passive waiting for successive stimulus presentations). These active brain/mind processes are central to human cognition and may be said to define our “natural cognition” much better than the narrow range of cognition-related behaviors (typically, ‘watch for’ and ‘press’) recorded during most previous and current brain imaging experiments.

3. Three challenges

Do current brain imaging methods allow observation and modeling of brain dynamics accompanying natural cognitive processes related to human motor behavior in a natural, dynamic 3-D environment? The answer is clearly no, in large part because of three obstacles inherent to now-standard brain imaging modalities.

3.1. Movement artifact

Brain activity during active full-body movement cannot be measured using most established brain imaging methods because their sensor apparatus is too heavy to be worn and carried so as to follow participant head movements. This is especially true for functional magnetic resonance imaging (fMRI), now the most often reported brain imaging modality in human neuroscience research. It is also the case for PET and magnetoencephalography (MEG). In these modalities, head movement may both add artifacts to while at the same time substantially modifying the recorded signals, creating alterations that cannot simply be identified and subtracted from the recorded signals to recapture the actual brain signals. Only modalities using smaller, portable or wearable sensors – electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) – are suitable for measuring brain activity supporting behaviors involving a normal range of head movement.

3.2. Movement speed

Metabolic brain imaging modalities including fMRI, PET, SPECT, and fNIRS measure local brain hemodynamic or other metabolic responses to the ever-changing brain energy use and needs. Their temporal resolution is limited by the speed of the underlying macroscopic hemodynamic or metabolic processes, which evolve over seconds. Meanwhile, the neural dynamics preceding, accompanying, and following human actions unfold on a broad millisecond to second time scale. As a consequence, brain imaging approaches measuring hemodynamic or other slow metabolic changes are inadequate for measuring brain dynamics supporting most natural cognition. Thus, the most promising means to record brain activity supporting natural cognition while avoiding the head stasis and temporal resolution problems is recording brain electrical activity with high temporal resolution using high-density lightweight scalp EEG electrodes.

Though even most commercial high-density EEG sensor systems are still tethered by wires to the subject and to a recording computer, they should have long allowed measurements of brain electrical activity during a wider range of active behaviors. Yet a large majority of EEG experimental protocols still typically restrict all movements by the participants excepting finger key presses that are in turn treated as if they have ‘negligible’ extent and duration — effectively, motoric point processes. This restriction is because movements of the head and eyes, and by implication, possibly in other parts of the body as well, may be accompanied by electromyographic (EMG) activity and corneo-retinal electrooculographic (EOG) potentials, which were summed with brain EEG signals of interest in scalp electrode recordings and were long thought difficult or impossible to separate cleanly from other ongoing EEG scalp signals (Makeig et al., 1996, 2009).

Previous and still most current data analysis approaches to modeling EEG brain activity have focused on analysis of the individual scalp-recorded channel signals. In EEG recordings, many sources of brain and non-brain electrical activity arrive at each electrode by volume conduction and are summed there to form a varying electrode potential; each EEG channel signal actually records the time-varying difference between the summed electrode potentials at two electrodes (or sometimes, sets of electrodes). Considered separately, each channel signal does not contain information allowing the researcher to separate brain from non-brain activity contributing to it. Thus, in most EEG studies any extensive participant movement has been considered a source of corrupting artifact (movement varying sensor/scalp interface, sensor cable, line noise, etc.), and so EEG recording protocols typically require participants to sit or lie still while deliberately inhibiting all movements.

Brain and non-brain source separation is possible only by: 1) identifying and removing individual artifacts from individual channel signals, or 2) by considering the mass of recorded channel data together. As the number of artifact types is large and all can be expected to have sample variability, the first option cannot be completely successful. However, the second approach, which amounts to performing spatial filtering (or even spatiotemporal filtering) on the whole data, has proven to be surprisingly successful in most instances, as we discuss below.

3.3. Movement and cognition

The restrictions on participant movements in typical brain imaging protocols do not appear to pose problems for neuroimaging studies investigating brain dynamics associated with physically ‘detached’ human cognition — for example, the cogitation of Rodin's sculptured character, ‘The Thinker,’ or of a book reader or movie viewer absorbing a presented tale. However, the cognitive processes observed in studies involving passive viewing of presented depictions of objects or symbols may not be identical to those elicited when participants carry out tasks involving motivated physical actions (and interactions) in a more complex 3-D environment.

There may likely be a good deal of overlap between brain dynamics supporting actual behavior and behavior perceived in others, even via a projected movie or television show. The brain's ‘mirror’ system is now thought to perceive movements by others ‘as if’ the perceiver was making the same movement (Rizzolatti and Craighero, 2004), and the concept of embodied cognition embraces the idea that our thinking and even language about abstractions such as math are built on ‘as if’ movement by the perceiver. For example, think of the phrase “a long time in the future,” in which the passage of time is conceptualized by analogy to movement across a ‘long’ distance.

An expected difference between imagined and actual movement is more readily evident for tasks involving complex motor behavior. One example is spatial orientation in which idiothetic information from muscles, joints, and the vestibular system influences how a navigator represents the spatial environment that they are moving in (Klatzky et al., 1998; Gramann, 2013). The cognitive processes and associated brain dynamics are directly influenced by active behavior of the navigator. Thus, the resulting brain dynamics accompanying natural cognition are likely to differ from those observed when participants are not allowed to move.

4. Natural cognition is coupled to active behavior

From the problems described above it is clear that the restriction of active behavior in established brain imaging studies impacts investigations of natural cognition. Human cognitive processes are based on our modes of use of our physical structure in our natural environment (Wilson, 2002) and support motor control in concert with perception (Churchland et al., 1994). As a consequence, analyzing human brain activity in combination with active motor behavior could reveal important new insights into the brain dynamics supporting human cognition.

Two yet unconnected strands of research emphasize the tight coupling of behavior and cognition and as a consequence, the coupling of behavior and brain dynamics. These are embodied cognition research and investigations of neuron-scale brain activity in behaving animals.

In embodied cognition research, the connection of behavior to the given environment and behavioral context is stressed so as to influence cognitive processing (Wilson, 2002). Examples are the influence of action plans on the perception of color or form when the action plans provide information for open parameters of that action (e.g., Wykowska et al., 2009) or the impairment of spatial orientation when movement-related idiothetic information about body rotation and translation is absent (e.g., Klatzky et al., 1998; Gramann et al., 2005; Plank et al., 2010; Gramann, 2013), and the demonstration of augmented retrieval of autobiographical memories via assumption of associated body postures (Dijkstra et al., 2007).

Investigations of neural firing patterns in different species of behaving animals demonstrate that brain dynamics may depend on the locomotor state of the animal (Maimon et al., 2010; Niell and Stryker, 2010). These results support the assumption that changes in behavioral state are accompanied by changes in brain dynamic state to allow adaptation to differences in incoming idiothetic and allothetic information (Gramann et al., 2011). While invasive recordings can measure brain dynamic states in stationary animals and recently also, in behaving animals (Schulz and Vaska, 2011), for ethical reasons this approach is not an option for research in healthy humans.

A fundamental question posed by these research areas, however remains: Are human brain dynamics both shaped by and dependent on active behavior? This question has been little investigated because of the technical and analytic limitations described above.

5. How to image natural cognition

If the brain dynamics and associated cognitive processes accompanying motor behavior are shaped by and to some extent vary with that behavior, important aspects of brain dynamic organization may not yet have been observed using static brain imaging modalities and paradigms. To overcome the methodological restrictions of established brain imaging approaches, we are developing a mobile brain/body imaging (MoBI) modality based on synchronous recording of high-density EEG with body motion capture and eye gaze tracking, plus other physiological measures, while participants perform motivated behaviors in three-dimensional environments (Makeig et al., 2009; Gramann et al., 2010, 2011; Gwin et al., 2010, 2011). Fig. 1 gives examples of different technical setups of current research using MoBI technology.

Fig. 1.

Fig. 1

Open in a new tab

Examples for MoBI setups from left to right (A and B) as used in the Mobile Brain/Body Imaging Lab at the Swartz Center for Computational Neuroscience (Makeig), (C) the Human Neuromechanics Laboratory in Michigan, and (D) the Berlin Mobile Brain/Body Imaging Lab in Berlin (Gramann). (A) In a ‘conducting’ experiment, novice and expert music listeners were invited to expressively ‘conduct’ music excerpts while their movements and EEG were recorded. Here a participant with 128-channel EEG cap and full-body motion capture suit with an addition LED sensor on the middle finger of his ‘conducting’ hand (picture courtesy Dr. Grace Leslie). (B) A dart game investigation with a participant aiming at the center of the dart board and throwing a dart. Here the recording included 128 EEG electrodes, 64 electrodes measuring neck muscle activity, and 64 arm electrode, motion capture, ground force plate, video, and behavioral measures (picture courtesy Dr. Makoto Miyakoshi). (C) A gait research setup with a participant on a treadmill, 128 Channel EEG, motion capture of the lower limbs, EMG of the lower limbs, a dual band force measuring treadmill, and external input devices for manual reactions. (D) A participant wearing 128-channel EEG, 32 channels for recording neck muscle activity, plus motion capture reflectors on the head, upper torso, and finger while playing a flying sphere game.

As evident from Fig. 1, current MoBI experiments allow participants to move in a relatively free manner. However, movement restriction due to cables from EEG, motion capture, and possibly other recording devices hamper natural behavior to a certain degree. Thus, the new MoBI approach both requires and is stimulating the development of new technologies for recording brain electrical activity and behavior to allow for absolute natural behavior without movement restrictions, new software solutions for synchronous multimodal recording and visualization, and new approaches to analyzing multiple streams of physiological and behavioral data.

6. Sensor technology

Conventional experimental EEG setups use wet Ag/AgCl electrodes that generally provide good signal quality in laboratory and clinical recordings (Thakor, 1999). In the case of high-density EEG recordings, connecting electrodes to the scalp using conductive gel can lead to short circuits between proximal electrodes (Roberto, 2010) and degradation of signal quality over longer measurement periods may be observed (Ferree et al., 2001). Another important restriction arises from running wires from each of the electrodes to the recording device. In laboratory experiments in which the participant is asked not to move, this is not a problem. In MoBI experiments featuring participant movements, however, cables inevitably restrict the movement range of participants and may well introduce mechanical artifacts (e.g., cable sway; Gramann et al., 2010; Gwin et al., 2010). While a few commercial high-density EEG systems are sufficiently light-weight to be carried by the participant in a small backpack, future MoBI experiments will be able to use new and now rapidly evolving wireless EEG and other sensor technology to allow participants ever more complete mobility while reducing recording artifacts associated with participant movements (Griss et al., 2002; Ko et al., 2006; Ruffini et al., 2008; Chi et al., 2010; Lin et al., 2011; Debener et al., 2012).

7. MoBI recording and analysis software

To allow investigation of brain dynamics during active participant behavior, data from modalities including EEG, eye tracking, and body motion capture, need to be recorded synchronously and then jointly analyzed. The development of adequate software to process the increasing amounts of synchronously recorded data so as to enable scientists to explore relationships between behavior and brain dynamics is a necessary and key aspect for developing a productive mobile brain/body imaging (MoBI) modality. This challenge is addressed at the Swartz Center for Computational Neuroscience of the University of California San Diego by, first, developing a software framework that also allows near-real time computations on data streams to affect the experimental protocol and stimulation. Our experimental real-time interactive control and analysis (ERICA) framework has evolved over several years and will doubtless continue to evolve to meet evolving MoBI data collection, visualization, and analysis needs. Currently, its key constituents are several software packages: LSL, XDF, ESS, SNAP, HED, EEGLAB, and MoBILAB.

7.1. LSL data collection

The Laboratory Streaming Layer (LSL) framework (code.google.com/p/labstreaminglayer) manages data collection in experiments involving concurrent recording via different hardware systems. LSL drivers for laboratory devices receive data transported across a local area network (LAN) using the UDP protocol to collect the data on one or more LAN computers. LSL then can save the data streams with time markers allowing later joint analysis of synchronous phenomena in more than one stream, make the data streams available for near real-time computation, and/or visualize the data for better experiment control and supervision. So far drivers have been written for several popular EEG systems, eye trackers, and motion capture systems, plus a range of devices including the Wii controller, a ground force measuring system and video and audio recording (Fig. 2).

Fig. 2.

Fig. 2

Open in a new tab

Schematic view of the Lab Streaming Layer (LSL) software framework for collecting, storing, and processing multi-modal laboratory data including data collected in MoBI experiments. LSL runs on a local area network (or, conceptually, a compute cloud network) and efficiently links data providers (physiological and/or behavioral recording systems) with data consumers (data viewer, recorder, or analysis facilities) in MoBI experiments.

The efficiency of LSL coding allows support for highly complex recording schemes linking computers running Windows, Mac OS, and Linux 64-bit or 32-bit operating systems. When needed, device drivers for new recording systems are often simple to write. Although the efficient, low-level LSL code tags incoming data samples with accurate time-of-arrival, delays between recording system data input (EEG, video, etc.) and LSL reception must be observed, computed, and used in the analysis process to obtain maximum accuracy in inter-stream timing. Luckily, for many recording systems these delays are relatively fixed, though e.g. sub-millisecond timing accuracy between streams may not be obtainable in many cases from a LAN-based system such as LSL.

7.2. XDF data storage

So far no EEG data format has received universal support. Further, current open EEG data formats were not constructed with concurrent collection of multimodal data streams in mind. Therefore, we have formalized an open, highly extensible data format, XDF (for Extensible Data Format (http://code.google.com/p/xdf), that is intended as a community-built and maintained format for storage and analysis of all types of laboratory physiological and behavioral data. LSL has routines for saving multi-stream data in XDF format.

7.3. ESS data description

To allow automated analysis and meta-analysis, LSL implements adding XDF header information about the data recording systems used and their parameters, the participant code and task, and other experimental conditions. Incorporating as much of this information as possible in the original stored XDF data collection makes it unlikely to ever be separated from the stored or archived data. To allow automated use of this data, an Experimental Study Schema (ESS) data description language is being developed (Fig. 3).

Fig. 3.

Fig. 3

Open in a new tab

Architecture of the Simulation and Neuroscience Application Platform (SNAP). Users create SNAP scripts that run desired experimental protocols (top panel). SNAP component functions run on top of and interact with the core Panda3D game engine. SNAP itself runs on the open computer language Python. SNAP allows relatively simple Python scripting of a wide range of fixed or interactive task paradigms, while also supporting development and delivery of highly complex, video game-like MoBI experiment applications.

7.4. SNAP experiment control

The SNAP (Simulation and Neuroscience Application Platform) environment can run experimental protocols involving one or more stimulation modalities that may incorporate near-real time analysis of one or more LSL data streams to allow interactive features. SNAP was built on top of the open source Panda3D game engine (panda3d.org) and uses Python as its primary scripting language. SNAP allows relatively simple, script-level development of complex, interactive experimental paradigms in which, for example, sensory feedback depends on one or more participants’ body locations, pointing directions, and/or eye gaze paths.

7.5. HED event tagging

To allow automated data analysis and meta-analysis of MoBI experiment data, key experimental events and their exact times of occurrence must be recorded and described. The Hierarchical Event Descriptor (HED) event-tagging system currently under development provides a common basis for building a hierarchical specification of a wide range of experimental events. Using appropriate SNAP script commands, HED tags may be incorporated into XDF data during data collection. To provide HED tags to pre-recorded data, a Java application and associated EEGLAB- and MoBILAB-compatible functions are being developed at the University of Texas San Antonio.

7.6. MoBILAB data analysis

EEGLAB (Delorme and Makeig, 2004), an open source environment for electrophysiological data analysis running on Matlab (The Mathworks, Inc.), is currently the most widely used analysis environment for electrophysiological data analysis for cognitive neuroscience according to a recent survey (Hanke and Halchenko, 2011). EEGLAB also supports plug-in functions and toolboxes that automatically appear in the EEGLAB graphic user interface (gui) of users who download them. To date, at least 30 EEGLAB plug-ins have been made available, many of them complex and elaborate toolboxes supporting a wide range of data analysis and visualization approaches.

However, EEGLAB, was originally designed for analysis of standard EEG experiment data. Thus its internal structure cannot be readily extended to support analysis of multi-modal data. Therefore a new Matlab toolbox, MoBILAB, is now under development (http://sccn.ucsd.edu/wiki/Mobilab_software). Written in object-oriented Matlab, MoBILAB can read MoBI data collected in XDF format (e.g., using LSL), and can visualize and operate efficiently on one or more of its constituent data streams. The centerpiece of the MoBILAB user environment is its multistream viewer that allows animated or manually advanced inspection of multiple concurrent MoBI data streams (EEG, motion capture, video, audio, etc.) plus a facility to allow inspection-based annotation or event marking. Tools for analysis of motion capture position marker data are currently available, as are tools for exporting selected and annotated portions of the EEG data to EEGLAB for EEG-centered analyses.

7.7. Data analysis approach

An early target for MoBI analysis of EEG data is to identify the timing and nature of motor decision events via changes in body movement (e.g., movement starts and stops, or course alterations). These can be identified as local maxima in acceleration (second-derivative) or jerk (third-derivative) magnitude time series of a participant limb or motion capture marker trajectory. To identify such motion events, one must properly low-pass filter the motion capture data and then identify (and carefully validate) local maxima in the jerk time series. MoBILAB includes facilities for doing this. Once one or more classes of motor decision events are identified in the behavioral data, then standard analysis of EEG data epochs surrounding the events of interest may be performed using EEGLAB, either on the natural time-locked EEG epochs or after time warping the epochs to normalize the duration of one or more movement phases across epochs. More advanced analysis approaches allowing for delays between brain dynamic features and behavioral events will be required to more fully model MoBI data. MoBILAB is designed to be readily extensible to support new approaches, including via a plug-in facility.

8. Data analyses

Established cognitive experiments record EEG while subjects are seated in a dimly lit and sound attenuated room, waiting for stimuli to be presented, without moving any part of their body or even their eyes. Reactions to presented stimuli are usually restricted to single button presses or minimal movements of the feet or hands. The suppression of eye movements, or any other movements of the body, avoids the relatively strong electrical potentials that are associated with movement of the eyes or contractions of neck or superficial skull and facial muscles (Makeig et al., 2009). Time periods with electromyographic or ocular activity are typically removed offline by rejecting the contaminated signal or by trying to regress out prototypic artifacts. The recorded signals are then epoched relative to the onset of a stimulus or class of stimuli and averaged over all epochs, assuming that activity unrelated to stimulus processing will be averaged out.

The restriction of participant movements in EEG investigations is thus primarily based on the fact that, due to volume conduction, non-brain related activity will contaminate the signal of interest. However, movement of the eyes or contraction of muscles and the accompanying proprioceptive feedback reflect active cognition and impact information processing (Biguer et al., 1988; Bove et al., 2002; Hayhoe and Ballard, 2005).

9. Independent component analysis (ICA) for mobile brain imaging

Spatial filtering based on the information content of the signals can be used to solve the problem of mixing of source signals at the electrodes by volume conduction (Makeig et al., 1996). Independent component analysis (ICA), a linear decomposition approach, separates multichannel data into independent component (IC) activities. Each IC activity is maximally statistically independent from any other IC activity and differs with respect to the relative strengths and polarities of its volume-conducted activity at the sensors. The presumption that ICA decomposition separates EEG data into physiologically and (very often) functional distinct sources is based on model assumptions including that the sources are spatially fixed throughout the data and the number of independent sources is equal to or less than the number of scalp sensors. In practice, when applied to a large enough amount of stable data, ICA can separate out activities of dozens of maximally independent information sources whose scalp maps near perfectly fit single-dipole (‘dipolar’) projections expected of cortical EEG sources representing synchronous activity across a highly connected cortical patch (Jung et al., 2000; Makeig et al., 2004; Gramann et al., 2010; Delorme et al., 2012) (Fig. 4).

Fig. 4.

Fig. 4

Open in a new tab

A) Grand average event-related potentials (ERPs) and single subject ERPs before and after spatially filtering EEG data. Participants were fast walking on a treadmill while detecting visually presented targets. Overlapping single-subject ERP traces are shown before (light pink traces) and after (gray traces) spatially filtering and rejection of artifacts using ICA. Bold traces show the grand average ERPs at the indicated electrode locations in the fast walking condition, before (red) and after (black) removing non-brain independent component (IC) processes. Scalp maps show grand average scalp topographies of the raw (left) and the artifact-removed ERPs (right) at 400 ms. White dots indicate the locations of the indicated electrodes. B) Upper and lower rows display scalp maps from mean projections to the scalp of the indicated clusters of independent component (IC) processes. Upper row from left to right displays scalp maps of brain-based clusters with cluster centroid dipole locations located in or near the anterior cingulate cortex, right and left motor cortex, and superior parietal cortex. Middle row displays equivalent-dipole locations of IC processes (small spheres) and respective IC cluster centroids (large spheres) projected on horizontal, sagittal, and coronal views of the standard MNI brain. (Yellow) Neck-muscle ICs; (gray) eye-movement ICs; (other colors) brain-based ICs. Lower row from left to right displays scalp maps of non-brain-based clusters with cluster centroid dipole locations located in or near the neck region reflecting neck muscle activity (left splenius capitis and right sternocleidomastoid) and vertical and horizontal eye movement activity.

Modified from Gramann et al. (2010).

Single equivalent current dipole models can thus be used to locate the origins of temporally-independent sources in the physical brain space. The accuracy of the position estimate depends on the accuracy of the electrical forward head model used in the process. As a consequence, reconstructed source locations should be considered spatial approximations of the centers of the unknown cortical source patches. New methods however, demonstrate significant improvements in source reconstruction accuracy, particularly when accurate electrode positions are obtained and (more so) when realistic head models built from structural MR scans are used (Akalin Acar and Makeig, 2013).

With respect to actively behaving participants, it should be pointed out that ICA separates and dissociates the contribution of brain and non-brain sources including mechanical artifacts (e.g., line noise), as well as biological signals that are important for cognitive processing — including eye movements and muscle activities (Jung et al., 2000; Gramann et al., 2010). The very nature of MoBI recordings, involving moving participants and concomitant EMG and eye movement activity, requires new approaches to handling of non-brain (‘artifactual’) activity. While this is an important issue, it is beyond the scope of this review to go into the details of these approaches. The interested reader is referred to the papers by Gwin et al. (2010, 2011), Lau et al. (2012) and Safieddine et al. (2012).

10. Applications

Previous work in our laboratories demonstrates that MoBI can be used to investigate the neural underpinnings of visual attention while participants actively walk with different speeds (Gramann et al., 2010). Analyses of functional brain activity are possible even when participants are running on a treadmill (Gwin et al., 2010). Further, brain dynamics underlying active orientation towards objects in the environment reveal a tight coupling of brain electrical activity with specific aspects of the orienting movement. These early studies are a proof of principle and several laboratories are now investigating cortical activity during active movement of participants including treadmill walking (Presacco et al., 2012), control of robotic exoskeletons for gait rehabilitation (Do et al.), or auditory attention during active behavior after cochlear implants (Debener et al., 2012). Further development of the MoBI approach and improved technologies for mobile and wireless sensing of brain activity and movements will provide new insights into brain dynamics underlying natural cognition that will lead to new applications. In the near future, the following research questions will likely be addressed using the MoBI approach.

10.1. Gait research and gait rehabilitation

There is a strong need to develop feedforward controllers for robotic rehabilitation devices. Combining robust electrocortical signals via wearable, dry-electrode EEG with detailed behavioral information via wearable, unobtrusive inertial motion capture will likely lead to the development of clinically relevant devices for gait retraining and intelligent prosthetic support systems that respond to user intent.

10.2. Neuroergonomics

Innovations in human–machine-interface design involving more natural human communication abilities (hand/arm and facial gestures, speech, voice inflection, etc.) are now occurring with increasing pace. The vision of combining these with active appreciation for the cognitive state, response, and intent of the operator, learned directly from combined wearable, dry-electrode EEG and complete behavior capture, is being studied explicitly in many laboratories and is likely to have many applications in diverse workplaces.

10.3. Computer-based training

Individualized training programs that use brain/behavioral data to assess, continually and in near real-time, the cognitive state, reactions, and intent of users could become more efficient, by far, than applications that have no knowledge of user state or intent.

10.4. Developmental psychology

Development of cognitive abilities, expressed in action, in infants and children is a clear opportunity for MoBI-based study. For example, Liao and colleagues investigated cortical brain dynamics of 3-year old children playing a game with their mothers using a touch screen while their EEG and body movements are monitored (Liao et al., 2012). Studies of movement decisions in normal aging are clearly possible and can give new information about changes in natural active cognition during aging.

10.5. Neurology

Many neurologic disorders involve changes in motor abilities and decision-making, including Parkinson's, Alzheimer's, and autism. MoBI methods can be used to develop new paradigms to study these abnormalities. For example, high functional autistic participants require more time to change their direction of movement compared to matched controls. MoBI can now be used to study what EEG dynamic differences underlie these behavioral differences. A MoBI study of the dynamics of brain electrical activity and electrically and behaviorally recorded Parkinson tremor induced by turning off an implanted deep brain stimulator (DBS) device is also underway, as is a study of recovery from stroke using active stimulus selection by patients with EEG recording.

10.6. Psychiatry

Most psychiatric illness is still diagnosed only on the basis of clinical interviews, which are difficult to make objective. However, behavioral abnormalities are clear in depression, schizophrenia, Tourette's, and other disorders. Here, MoBI studies can open new windows into links between altered brain dynamics and behavior.

10.7. Spatial orientation

Brain dynamics associated with active orienting movements and processing of idiothetic information are almost unstudied. No experiments have yet been able to reveal the influence of vestibular and proprioceptive information processing on human brain activity during spatial orientation or other spatial tasks including whole body movements.

11. Perspective

Our work demonstrates that it is possible to analyze brain dynamics accompanying active cognition. A framework to support new directions in experimental protocols, integration of different data streams, and analyses approaches is being developed to support users in overcoming the restrictions of established brain imaging methods and to investigate the full range of natural cognition. First investigations show the tight coupling of brain dynamics and active behavior and provide a first insight into the possibilities and the potential of this exciting new research field. The greatest obstacle that currently presents itself is the need for better data mining tools for interpreting large data sets available with MoBI approaches.

Footnotes

☆

This work was supported in part by the Office of Naval Research to D.F. (N000140811215), the Army Research Laboratory to D.F. (W911NF-09-1-0139 & W911NF-10-2-0022), an Air Force Office of Scientific Research National Defense Science and Engineering Graduate Fellowship to J.G. (32 CFR 168a), the National Institutes of Health to D.F. (R01 NS073649) and S.M. (S.M., 5R01 NS047293-08), and a gift to UCSD from the Swartz Foundation (Old Field NY). The content is solely the responsibility of the authors and does not necessarily represent the official views of ONR, ARL, AFOSR, or NIH.

References

  1. Akalin Acar Z, Makeig S. Effects of forward model errors on EEG source localization. Brain Topogr. 2013;26:378–396. doi: 10.1007/s10548-012-0274-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Biguer B, Donaldson IM, Hein A, Jeannerod M. Neck muscle vibration modifies the representation of visual motion and direction in man. Brain. 1988;111(Pt 6):1405–1424. doi: 10.1093/brain/111.6.1405. [DOI] [PubMed] [Google Scholar]
  3. Bove M, Courtine G, Schieppati M. Neck muscle vibration and spatial orientation during stepping in place in humans. J. Neurophysiol. 2002;88:2232–2241. doi: 10.1152/jn.00198.2002. [DOI] [PubMed] [Google Scholar]
  4. Chi YM, Jung TP, Cauwenberghs G. Dry-contact and noncontact biopotential electrodes: methodological review. IEEE Rev. Biomed. Eng. 2010;3:106–119. doi: 10.1109/RBME.2010.2084078. [DOI] [PubMed] [Google Scholar]
  5. Churchland PS, Ramachandran VS, Sejnowski TJ. A Critique of Pure Vision. In: Koch C, Davis JL, editors. Large-scale Neuronal Theories of the Brain. MIT Press; Cambridge, MA: 1994. pp. 23–60. [Google Scholar]
  6. Corbetta M, Shulman GL. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 2002;3:201–215. doi: 10.1038/nrn755. [DOI] [PubMed] [Google Scholar]
  7. Damasio AR. The somatic marker hypothesis and the possible functions of the pre-frontal cortex. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1996;351:1413–1420. doi: 10.1098/rstb.1996.0125. [DOI] [PubMed] [Google Scholar]
  8. Debener S, Minow F, Emkes R, Gandras K, De Vos M. How about taking a low-cost, small, and wireless EEG for a walk? Psychophysiology. 2012;49:1617–1621. doi: 10.1111/j.1469-8986.2012.01471.x. [DOI] [PubMed] [Google Scholar]
  9. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods. 2004;134:9–21. doi: 10.1016/j.jneumeth.2003.10.009. [DOI] [PubMed] [Google Scholar]
  10. Delorme A, Palmer J, Onton J, Oostenveld R, Makeig S. Independent EEG sources are dipolar. PLoS ONE. 2012:7. doi: 10.1371/journal.pone.0030135. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dijkstra K, Kaschak MP, Zwaan RA. Body posture facilitates retrieval of autobiographical memories. Cognition. 2007;102:139–149. doi: 10.1016/j.cognition.2005.12.009. [DOI] [PubMed] [Google Scholar]
  12. Do AH, Wang PT, King CE, Chun SN. Brain-Machine-Body Interfaces Workshop. San Diego, CA: 2012. Nenadic Z Brain-Computer Interface Controlled Robotic Gait Orthosis. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ferree TC, Phan L, Russell GS, Tucker DM. Scalp electrode impedance, infection risk, and EEG data quality. Clin. Neurophysiol. 2001;112:536–544. doi: 10.1016/s1388-2457(00)00533-2. [DOI] [PubMed] [Google Scholar]
  14. Gabrieli JDE, Brewer JB, Desmond JE, Glover GH. Separate neural bases of two fundamental memory processes in the human medial temporal lobe. Science. 1997;276:264–266. doi: 10.1126/science.276.5310.264. [DOI] [PubMed] [Google Scholar]
  15. Goodale MA, Milner AD. Separate visual pathways for perception and action. Trends Neurosci. 1992;15:20–25. doi: 10.1016/0166-2236(92)90344-8. [DOI] [PubMed] [Google Scholar]
  16. Gramann K. Embodiment of spatial reference frames and individual differences in reference frame proclivity. Spat. Cogn. Comput. 2013;13:1–25. [Google Scholar]
  17. Gramann K, Muller HJ, Eick EM, Schonebeck B. Evidence of separable spatial representations in a virtual navigation task. J. Exp. Psychol. Hum. 2005;31:1199–1223. doi: 10.1037/0096-1523.31.6.1199. [DOI] [PubMed] [Google Scholar]
  18. Gramann K, Gwin JT, Bigdely-Shamlo N, Ferris DP, Makeig S. Visual evoked responses during standing and walking. Front. Hum. Neurosci. 2010;4:202. doi: 10.3389/fnhum.2010.00202. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gramann K, Gwin JT, Ferris DP, Oie K, Jung TP, Lin CT, Liao LD, Makeig S. Cognition in action: imaging brain/body dynamics in mobile humans. Rev. Neurosci. 2011;22:593–608. doi: 10.1515/RNS.2011.047. [DOI] [PubMed] [Google Scholar]
  20. Griss P, Tolvanen-Laakso HK, Meriläinen P, Stemme G. Characterization of micromachined spiked biopotential electrodes. IEEE Trans. Biomed. Eng. 2002;49:597–604. doi: 10.1109/TBME.2002.1001974. [DOI] [PubMed] [Google Scholar]
  21. Gwin JT, Gramann K, Makeig S, Ferris DP. Removal of movement artifact from high-density EEG recorded during walking and running. J. Neurophysiol. 2010;103:3526–3534. doi: 10.1152/jn.00105.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Gwin JT, Gramann K, Makeig S, Ferris DP. Electrocortical activity is coupled to gait cycle phase during treadmill walking. Neuroimage. 2011;54:1289–1296. doi: 10.1016/j.neuroimage.2010.08.066. [DOI] [PubMed] [Google Scholar]
  23. Handy TC, Grafton ST, Shroff NM, Ketay S, Gazzaniga MS. Graspable objects grab attention when the potential for action is recognized. Nat. Neurosci. 2003;6:421–427. doi: 10.1038/nn1031. [DOI] [PubMed] [Google Scholar]
  24. Hanke M, Halchenko YO. Neuroscience runs on GNU/Linux. Front. Neuroinform. 2011;5:8. doi: 10.3389/fninf.2011.00008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Hayhoe M, Ballard D. Eye movements in natural behavior. Trends Cogn. Sci. 2005;9:188–194. doi: 10.1016/j.tics.2005.02.009. [DOI] [PubMed] [Google Scholar]
  26. Iacoboni M, Woods RP, Brass M, Bekkering H, Mazziotta JC, Rizzolatti G. Cortical mechanisms of human imitation. Science. 1999;286:2526–2528. doi: 10.1126/science.286.5449.2526. [DOI] [PubMed] [Google Scholar]
  27. Jung TP, Makeig S, Humphries C, Lee TW, McKeown MJ, Iragui V, Sejnowski TJ. Removing electroencephalographic artifacts by blind source separation. Psychophysiology. 2000;37:163–178. [PubMed] [Google Scholar]
  28. Klatzky RL, Loomis JM, Beall AC, Chance SS, Golledge RG. Spatial updating of self-position and orientation during real, imagined, and virtual locomotion. Psychol. Sci. 1998;9:293–298. [Google Scholar]
  29. Ko L-W, Chiou J-C, Lin C-T, Hong C-T, Jung T-P, Liang S-F, Jeng J-L. Using novel MEMS EEG sensors in detecting drowsiness application; Proceeding of IEEE Biomedical Circuits and Systems Conference (BioCAS 2006); London, U.K.. 2006. pp. 33–36. [Google Scholar]
  30. Lau TM, Gwin JT, McDowell KG, Ferris DP. Weighted phase lag index stability as an artifact resistant measure to detect cognitive EEG activity during locomotion. J. Neuroeng. Rehabil. 2012;9:47. doi: 10.1186/1743-0003-9-47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Liao Y, Danley J, Vankov A, Makeig S, Deak GO. Adult-like changes in mu rhythm activity in young children accompany action observation and execution in a social context; Meeting of the Society for Neuroscience New Orleans.2012. [Google Scholar]
  32. Lin CT, Liao LD, Liu YH, Wang IJ, Lin BS, Chang JY. Novel dry polymer foam electrodes for long-term EEG measurement. IEEE Trans. Biomed. Eng. 2011;58:1200–1207. doi: 10.1109/TBME.2010.2102353. [DOI] [PubMed] [Google Scholar]
  33. Maimon G, Straw AD, Dickinson MH. Active flight increases the gain of visual motion processing in Drosophila. Nat. Neurosci. 2010;13:393–399. doi: 10.1038/nn.2492. [DOI] [PubMed] [Google Scholar]
  34. Makeig S, Muller MM, Rockstroh B. Effects of voluntary movements on early auditory brain responses. Exp. Brain Res. 1996;110:487–492. doi: 10.1007/BF00229149. [DOI] [PubMed] [Google Scholar]
  35. Makeig S, Debener S, Onton J, Delorme A. Mining event-related brain dynamics. Trends Cogn. Sci. 2004;8:204–210. doi: 10.1016/j.tics.2004.03.008. [DOI] [PubMed] [Google Scholar]
  36. Makeig S, Gramann K, Jung TP, Sejnowski TJ, Poizner H. Linking brain, mind and behavior. Int. J. Psychophysiol. 2009;73:95–100. doi: 10.1016/j.ijpsycho.2008.11.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Mishkin M, Ungerleider LG. Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys. Behav. Brain Res. 1982;6:57–77. doi: 10.1016/0166-4328(82)90081-x. [DOI] [PubMed] [Google Scholar]
  38. Niell CM, Stryker MP. Modulation of visual responses by behavioral state in mouse visual cortex. Neuron. 2010;65:472–479. doi: 10.1016/j.neuron.2010.01.033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Paus T, Petrides M, Evans AC, Meyer E. Role of the human anterior cingulate cortex in the control of oculomotor, manual, and speech responses: a positron emission tomography study. J. Neurophysiol. 1993;70:453–469. doi: 10.1152/jn.1993.70.2.453. [DOI] [PubMed] [Google Scholar]
  40. Plank M, Müller H, Onton J, Makeig S, Gramann K. Human EEG correlates of spatial navigation within egocentric and allocentric reference frames. In: Hölscher C, et al., editors. Spatial Cognition VII. Vol. 6222. Heidelberg; Springer Berlin: 2010. pp. 191–206. [Google Scholar]
  41. Presacco A, Forrester LW, Contreras-Vidal JL. Decoding intra-limb and inter-limb kinematics during treadmill walking from scalp electroencephalographic (EEG) signals. IEEE Trans. Neural Syst. Rehabil. Eng. 2012;20:212–219. doi: 10.1109/TNSRE.2012.2188304. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, Shulman GL. A default mode of brain function. Proc. Natl. Acad. Sci. U. S. A. 2001;98:676–682. doi: 10.1073/pnas.98.2.676. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Rizzolatti G. The mirror neuron system and its function in humans. Anat. Embryol. (Berl.) 2005;210:419–421. doi: 10.1007/s00429-005-0039-z. [DOI] [PubMed] [Google Scholar]
  44. Rizzolatti G, Craighero L. The mirror-neuron system. Annu. Rev. Neurosci. 2004;27:169–192. doi: 10.1146/annurev.neuro.27.070203.144230. [DOI] [PubMed] [Google Scholar]
  45. Roberto M. The electrode–skin interface and optimal detection of bioelectric signals. Physiol. Meas. 2010:31. doi: 10.1088/0967-3334/31/10/e01. [DOI] [PubMed] [Google Scholar]
  46. Ruffini G, Dunne S, Fuentemilla L, Grau C, Farr E, Marco-Pallar J, Watts PCP, Silva SRP. First human trials of a dry electrophysiology sensor using a carbon nano-tube array interface. Sensors Actuators A Phys. 2008;144:275–279. [Google Scholar]
  47. Safieddine D, Kachenoura A, Albera L, Birot G, Karfoul A, Pasnicu A, Biraben A, Wendling F, Senhadji L, Merlet I. Removal of muscle artifact from EEG data: comparison between stochastic (ICA and CCA) and deterministic (EMD and wavelet-based) approaches. EURASIP J. Adv. Sig. Pr. 2012;2012(1):1–15. [Google Scholar]
  48. Schulz D, Vaska P. The emerging discipline of behavioral neuroimaging. Rev. Neurosci. 2011;22:591–592. doi: 10.1515/RNS.2011.051. [DOI] [PubMed] [Google Scholar]
  49. Squire LR, McKee RD. Declarative and nondeclarative memory in opposition: when prior events influence amnesic patients more than normal subjects. Mem. Cognit. 1993;21:424–430. doi: 10.3758/bf03197173. [DOI] [PubMed] [Google Scholar]
  50. Squire LR, Zola-Morgan S. The medial temporal lobe memory system. Science. 1991;253:1380–1386. doi: 10.1126/science.1896849. [DOI] [PubMed] [Google Scholar]
  51. Thakor NV. The Measurement, Instrumentation, and Sensors Handbook. CRC Press; 1999. Biopotentials and electro-physiology measurement. pp. 1–20. Chapter 74. [Google Scholar]
  52. Wilson M. Six views on embodied cognition. Psychon. Bull. Rev. 2002;9:625–636. doi: 10.3758/bf03196322. [DOI] [PubMed] [Google Scholar]
  53. Wykowska A, Schubo A, Hommel B. How you move is what you see: action planning biases selection in visual search. J. Exp. Psychol. Hum. 2009;35:1755–1769. doi: 10.1037/a0016798. [DOI] [PubMed] [Google Scholar]

RESOURCES