Optical neuroimaging of spoken language

Abstract

In this review I introduce the historical context and methods of optical neuroimaging, leading to the modern use of functional near-infrared spectroscopy (fNIRS) and high-density diffuse optical tomography (HD-DOT) to study human brain function. In its most frequent application, optical neuroimaging measures a hemodynamically-mediated signal indirectly related to neural processing, similar to that captured by fMRI. Compared to other approaches to measuring human brain function, optical imaging has many advantages: it is noninvasive, frequently portable, acoustically silent, robust to motion and muscle movement, and appropriate in many situations in which fMRI is not possible (for example, due to implanted medical devices). Challenges include producing a full-brain field of view, homogenous spatial resolution, and accurate source localization. Experimentally, optical neuroimaging has been used to study phoneme, word, and sentence processing in a variety of paradigms. With continuing technical and methodological improvements the future of optical neuroimaging is increasingly bright.

Introduction

As cognitive neuroscientists we have at our disposal an overwhelming number of methods that provide insight into human brain function with varying degrees of spatial resolution, temporal precision, participant comfort, and acoustic noise. Here I introduce optical neuroimaging and summarize some of the main areas in which it has contributed to our understanding of speech and language processing.

It is worth noting at the outset that there are many types of optical neuroimaging systems. Although they rely on the same basic principles, the technical details influence the resolution of the data that can be collected, and often affect the nomenclature. I use “optical neuroimaging” as an umbrella term to cover all of these systems.

I have focused on providing a broad overview of the methods and applications of optical neuroimaging, necessarily leaving out many relevant studies due to space constraints: this paper is better suited to be a first step in reading about optical neuroimaging of language than a final word. Other reviews on the use of optical imaging in language provide many useful details regarding the history and methodological background of optical imaging (Ferrari & Quaresima, 2012; Hillman, 2007; Scholkmann et al., 2014), as well as comprehensive overviews of prior studies of speech and language (Dieler, Tupak, & Fallgatter, 2012; Quaresima, Bisconti, & Ferrari, 2012; Rossi, Telkemeyer, Wartenburger, & Obrig, 2012).

Historical Background

Modern optical imaging methods can be traced back to technological and theoretical developments occurring in the first half of the 20^th century (Chance, 1991). These include the ability to measure differential absorption of light of multiple wavelengths, and being able to quantify hemoglobin deoxygenation in the earlobe (an early version of a pulse oximeter, still commonly used in medical settings). Although the relative transparency of tissues that might enable in vivo imaging was noted in the 1970s (Jöbsis, 1977), the first experiments using optical imaging to look at human brain function were published in the early 1990s. These early studies investigated neural function using paradigms including mental arithmetic (Hoshi & Tamura, 1993), a mirror drawing task (Okada, Tokumitsu, Hoshi, & Tamura, 1993), visual stimulation (Kato, Kamei, Takashima, & Ozaki, 1993; Villringer, Planck, Hock, Schleinkofer, & Dirnagl, 1993), and verbal reasoning (Chance, Zhuang, Unah, Alter, & Lipton, 1993). Over the past 25 years, methodological developments have involved increasing the amount of brain tissue imaged, and improving spatial resolution, cortical localization, and statistical methodology.

The first language studies using optical neuroimaging occurred in the late 1990s, as did the first optical neuroimaging study of infants (Meek et al., 1998). Sakatini and colleagues (1998) performed a variety of speech tasks, including confrontation naming, with both healthy participants and patients who had experienced a stroke. The authors observed signal changes in left prefrontal cortex, corresponding roughly with the left inferior frontal gyrus (“Broca’s area”) that differed between aphasic patients and controls. Watanabe and colleagues (1998) investigated hemispheric lateralization during a word-generation task and found that optical neuroimaging was able to detect differential activity between the left and right hemispheres. In epilepsy patients, the laterality of these findings agreed with results from a Wada test, providing external validation for language dominance. In an early study of speech comprehension, Sato and colleagues (1999) had participants perform a dichotic listening task with tones, sentences or stories. They found increased activity for the story task in left superior temporal cortex. Recent studies have investigated a breadth of topics comparable to other cognitive neuroscience modalities, including (but certainly not limited to) categorical perception (Minagawa-Kawai, Mori, Furuya, Hayashi, & Sato, 2002), speech production (Hull, Bortfeld, & Koons, 2009), language lateralization (Bisconti, Di Sante, Ferrari, & Quaresima, 2012; Watson, Dodrill, Farrell, Holmes, & Miller, 2004), and resting state functional connectivity (White et al., 2009).

Optical neuroimaging: The methods

At a basic level, optical imaging involves shining light into a tissue with one or more sources, and measuring light output from the tissue using one or more detectors. Although this can be done with exposed tissue, here I focus on the more typical application in cognitive neuroscience involving noninvasive imaging through the scalp and skull. Most optical imaging relies on the principle of near-infrared spectroscopy (NIRS; in the context of functional brain imaging, fNIRS) to determine what type of material the light has passed through. Optical fibers coupled to the head shine light with wavelengths in the near-infrared spectrum (~650–1000 nm) into the head. Typically two or more wavelengths are used, which have different absorption constants depending on the type of tissue. Source encoding (i.e., flashing sources at different points in time and frequencies) can also be used to help differentiate light from various sources.

Sensors detect light exiting the head, which has passed through the skull and superficial cortex. Light entering the head will be scattered in the tissue, and some of this light is absorbed by chromophores such as hemoglobin (Hb). Helpfully, the spectrum of light absorbed by hemoglobin depends on whether the hemoglobin is oxygenated or not. Thus, incorporating light models that differentiate how light will be absorbed by blood with oxygenated and deoxygenated hemoglobin allows the estimation of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) signals, as well as total hemoglobin (HbT). These measurements thus reflect hemodynamic signals comparable to those observed in blood-oxygenation level dependent (BOLD) fMRI (T. J. Huppert, Hoge, Diamond, Franceschini, & Boas, 2006; G. Strangman, Culver, Thompson, & Boas, 2002). Though it is often assumed that the fMRI BOLD signal is most closely associated with changes in HbR (Toronov et al., 2001), strong correlations have also been observed between the fMRI signal and HbO and HbT contrasts (Eggebrecht et al., 2014), and the exact relationship between various Hb measures and the BOLD signal is still not entirely clear (Gagnon et al., 2012; V. Y. Toronov, Zhang, & Webb, 2007). Nevertheless, the multiple simultaneous optical signals provide the opportunity to differentiate the effects of a stimulus on focal changes in cerebral blood volume from changes in HbO and HbR, and thereby better inform the investigator about the underlying neural activity. Many optical imaging studies report HbO alone, for simplicity or because there is stronger signal in the HbO than the HbR measurement, as local changes in blood flow are much stronger than local changes due to altered metabolic activity (Fox, Raichle, Mintun, & Dence, 1988). In addition, poor signal in HbR signals can be exacerbated by poor choice of wavelengths and thereby suboptimal spectroscopy. Cases where the HbO and HbR signals diverge should be addressed with caution as they may imply poor optical-scalp coupling due to artifact or altered neurovascular coupling (e.g., due to a stroke or other vascular event).

The temporal resolution of optical neuroimaging is relatively good compared to other imaging approaches (such as fMRI), with sampling rates between 5–100 Hz. Of course, when the signal being sampled depends on a relatively slow hemodynamic response, the effective resolution with respect to the timecourse of neural activity is lower. Nevertheless, the higher temporal resolution may help with signal processing and artifact rejection (Griffanti et al., 2014; Salimi-Khorshidi et al., 2014), although such approaches are not yet widely used in optical neuroimaging.

The spatial resolution of optical neuroimaging is determined in part by the density of the sources and detectors (related to both the number of sources and detectors, and the extent of coverage). Figure 1a shows examples of three different optode arrangements that vary in the number of channels, and Figure 1b illustrates the effect of channel spacing on spatial resolution. Recent technological improvements have enabled optical systems with increasingly large fields-of-view (FOVs) and greater densities of sources and detectors (Eggebrecht, et al., 2014). To distinguish these high-density systems from traditional NIRS, they are frequently referred to as diffuse optical imaging (DOI) or high-density diffuse optical tomography (HD-DOT), although they rely on the same basic principles of near-infrared spectroscopy as traditional fNIRS. It is important to note that having a high-density layout is a necessary condition to perform tomographic reconstruction, but not a sufficient one, as there are complex reconstruction algorithms needed to successfully obtain tomographic images.

Figure 1.

(a) Examples of source and detector arrangements used by Toronov et al. (2000), Huppert et al. (2006), and Eggebrecht et al. (2014). (b) Effects of source and detector spacing on homogeneity of error and effective spatial resolution based on simulated data. Top: source and detector spacing for two example grids. Middle: Localization error, defined as the separation between the known target location and the centroid of the voxels reconstructed above half-maximum contrast. Bottom: Effective spatial resolution, defined as the diameter of the circle centered at each target position needed to enclose the response. Modified from White and Culver (2010).

Spatial localization is a challenging aspect of optical neuroimaging. For many systems, spatial location is restricted to the level of a source-detector measurement. Optodes are typically placed on the head with respect to surface landmarks. Statistical analysis can be done for each channel, and the approximate anatomical location of an observed effect can be inferred based on the area of the source-detector location on the scalp. However, there have been improvements in source localization brought about in part through the use of high-density arrays. At least some modern processing pipelines incorporate structural brain scans to generate multicompartment anatomically-informed light models, allowing localizing of signals onto the cortical surface (Custo et al., 2010; Ferradal, Eggebrecht, Hassanpour, Snyder, & Culver, 2014; Zhan, Eggebrecht, Culver, & Dehghani, 2012). Validation studies show relatively good within-subject agreement with known functional organization (e.g., visual hemifields) and generally good correspondence with fMRI (Eggebrecht, et al., 2014; Eggebrecht et al., 2012; Zeff, White, Dehghani, Schlaggar, & Culver, 2007). Because the source-localized images are composed of 4D data (3D volumes in space, repeated over time), they are similar in nature to timeseries images obtained in fMRI, and optical imaging data are increasingly being modeled using statistical approaches inspired by fMRI (Hassanpour et al., 2014; Theodore J. Huppert, Diamond, Franceschini, & Boas, 2009; G. E. Strangman, Zhang, & Zeffiro, 2009; Ye, Tak, Jang, Jung, & Jang, 2009).

Challenges and solutions for studying spoken language with optical imaging

There are a number of challenges that arise when performing optical imaging of brain activity. Although many are not unique to language processing, they are important for researchers conducting language studies to be aware of.

Field-of-view limitations

A recurring challenge for optical neuroimaging is the limited field of view resulting from two independent limitations. The first is that the amount of the cortical surface that can be imaged depends on the number (and density) of sources and detectors. Many optical systems are designed to be relatively portable, which restricts the amount of equipment that can be used (and thus the number of sources and detectors). Unfortunately, the limited field of view makes it impossible to get a full picture of the cortical activation during an experiment. Luckily, this restriction can be mitigated by using an increased number of sources and detectors (at the expense of device portability): more recent HD-DOT systems can cover significant portions of the cortical surface (Eggebrecht, et al., 2014), with the prospect of whole-brain coverage in the coming years.

A more difficult problem comes from the fact that light diffuses rapidly once it leaves an optical source. Thus, signal strength falls off rapidly with cortical depth, with reasonable sensitivity reaching only approximately 1 cm into the cortical surface (Eggebrecht, et al., 2014). Although higher optical intensity might in principle provide information on deeper structures, health concerns regarding using such light in vivo has generally precluded its use. In addition, although a more powerful light would likely improve the signal to noise ratio of the optical signal, it would not necessarily increase the depth resolution that is limited primarily by the scattering properties of tissue. Even though imaging the entire superficial cortical surface reveals a great deal about cortical organization, a full picture that includes deeper cortical tissue and subcortical structures requires converging evidence from other techniques.

Spatial resolution and source localization

As discussed above, two areas in which optical neuroimaging has traditionally lagged behind fMRI are its spatial resolution and source localization accuracy. The degree of challenge these represent for language research depends on the level of anatomical specificity needed to test a particular hypothesis. For example, if a study is conducted to see whether the frontal cortex is involved in a particular task, a positive result could be easily interpreted. On the other hand, it would be more difficult to draw conclusions about nearby subdivisions of cortical regions that may support different functions (Fedorenko, Duncan, & Kanwisher, 2012; Goucha & Friederici, 2015).

Equipment constraints

In many ways, optical brain imaging is less constraining than other modalities such as fMRI: the equipment is generally smaller, less expensive, and there are no magnetic or radioactive safety concerns. However, optical neuroimaging brings its own equipment challenges. These include discomfort from the optodes pressing on the head, managing the optical fibers, and the weight of the cap (which can be especially relevant when testing babies), all of which are more challenging in high-density systems due to the increased number of channels.

Key empirical contributions of optical imaging to spoken language processing

Optical neuroimaging has been used to study spoken language processing for many years. Examples of some of these findings, using different methodological approaches, are shown in Figure 2. Although it is impossible to do justice to all of the topics that have been studied, I focus on findings in three main areas that are particular strengths for optical imaging.

Figure 2.

Examples of optical neuroimaging studies of language. (a) Results from a single channel optical system imaging left prefrontal cortex while participants performed a naming task (Sakatani, et al., 1998). (b) Left-hemisphere multi-channel data from infants listening to sequences which differed in their repetition (ABC vs. ABB patterns) (Gervain, et al., 2008). The findings show increased responses for the ABB patterns, particularly for the HbO (magenta). Boxes indicate regions of interest made up of channels over left frontal and temporal cortices. (c) Source-localized HD-DOT results showing activity during spoken sentence comprehension in adults compared to an acoustic baseline (top) and for a syntactic complexity comparison between sentences with subject-relative and object-relative center-embedded clauses (bottom) (Hassanpour, et al., 2015).

Language development in infants and children

Portability, lack of acoustic noise, and resilience to movement have made optical neuroimaging especially appealing in studying brain function in infants and children (Ferradal et al., 2016; Gervain et al., 2011; Lloyd-Fox, Blasi, & Elwell, 2010). For example, many systems permit infants to sit in their parent or caretaker’s lap during testing, which can be particularly useful (and typically preferable to the swaddling approach taking in fMRI studies). Subsequent infant studies have investigated topics such as auditory processing (Taga & Asakawa, 2007; Zaramella et al., 2001), connected speech comprehension (Bortfeld, Wruck, & Boas, 2007), and prosody (Homae, Watanabe, Nakano, & Taga, 2007).

One important topic in language development is the age at which language functions begin to lateralize into a dominant hemisphere. In an early and influential study, Peña and colleagues (2003) presented 2–5 day old infants with forward and reversed speech (the same language the infants heard while in the womb). The forward speech produced a stronger response in the left hemisphere than in the right hemisphere, provocatively suggesting that even at this early age there were hints of language lateralization. Bortfeld et al. (2009) showed colorful visual stimuli to 6–9 month old infants either with a concurrent short story or with no story (visual-only). They found increased responses for the speech condition that were significantly larger in left temporal channels than in right temporal channels. These results are consistent with a lateralized response to spoken language by early on in postnatal development (Minagawa-Kawai et al., 2011; Sato et al., 2012).

A related question concerns the age at which speech-related processing develops (for example, sensitivity to speech-like sounds). Gervain and colleagues (2008) presented infants with speech segments containing a repeated syllable (that is, an ABB structure: “mubaba”) or no repetition (an ABC structure: “mubage”). The authors found evidence for increased activity in left temporal and inferior frontal regions that distinguished between the repeated and non-repeated syllable sequences (Figure 2b), suggesting the infants were able to distinguish between syllables, and were also sensitive to repetition.

Thus, optical neuroimaging has been instrumental in facilitating the study of newborn and infant language processing, and will likely continue to shed light on a number of very important areas of developmental cognitive neuroscience.

Spoken language processing in populations who are excluded from fMRI

Although not yet widely practiced, using optical imaging to measure brain function in patients who are not able to have an MRI is a critical contribution of optical neuroimaging to cognitive neuroscience. Patients with implanted medical devices, for example, are routinely excluded from MRI studies as a matter of course due to the potential for damage to the device or injury to the patient. Even with MR-compatible devices, image artifacts frequently make the data impossible to use. Furthermore, in many cases the devices also cause artifacts on MEG or EEG that affect data quality. (And in fact, the same logic would hold for non-implanted devices that participants might use during a study, such as a hearing aid.)

Fortunately, optical neuroimaging has no effect in implanted devices and is thus an appealing alternative. At the same time, the implanted devices have little effect on the optical signal outside of their immediate area. That is, data at detectors near the device will be affected because the device will physically be in the light path, but no systematic artifacts propagate throughout large portions of the dataset, as would occur with MEG or EEG. To date, optical imaging has been used in patients with deep brain stimulation (Murata et al., 2000) and cochlear implants (Bisconti et al., 2016; Pollonini et al., 2014; Saliba, Bortfeld, Levitin, & Oghalai, 2016; Sevy et al., 2010). In one study of cochlear implant recipients, activity in auditory cortex has been shown to correlate with behavioral word-report measures (Olds et al., 2016), suggesting optical neuroimaging may be able to provide an objective measure of speech intelligibility that does not depend on overt participant responses.

Imaging speech comprehension in quiet

One prominent advantage of optical imaging is that it is completely silent, thus avoiding the challenges caused by acoustic noise during fMRI (Peelle, 2014). Thus, although optical imaging is frequently chosen for other reasons (portability, special populations, etc.), it has special utility for studying speech comprehension. An advantage of all of the auditory and speech studies mentioned above is that, unlike similar studies conducted in fMRI, they reflect processing in a quieter acoustic environment more similar to that typically used in behavioral testing.

With increased field of view afforded by high-density optical imaging systems, distributed cortical networks can be simultaneously imaged and localized to the brain with a spatial resolution comparable to fMRI. Recent investigations into the cortical processing of language include single word processing (Eggebrecht, et al., 2014) and spoken sentence processing (Hassanpour, Eggebrecht, Culver, & Peelle, 2015), the latter illustrated in Figure 2c. To date, the results from studies using high-density systems have been largely confirmatory, with optical imaging producing results largely consistent with previous fMRI and PET experiments. However, these initial validation studies set the stage for more interesting extensions in future work, free of the challenges of acoustic MRI scanner noise.

Future directions

Although optical neuroimaging has already proven to be a valuable tool in cognitive neuroscience, future improvements have the potential to even further expand its utility for illuminating the neural basis of spoken language processing:

1. Increasing the field of view. Although imaging deep tissue is an inherent limitation of the technique, routinely obtaining data across the entire cortical surface would permit the simultaneous imaging of multiple cortical systems.

2. Increasing the density of sources and detectors. The accuracy of spatial localization is a concern for any researcher interested in the neuroanatomical basis for cognitive processing; high-density arrays provide superior (and more homogenous) localization.

3. Developing high-density systems that are portable. A drawback of current high-density systems is their lack of portability; developing high-density systems that retain the portability advantages of traditional sparse NIRS arrays would incorporate the best of both worlds.

4. Continuing to take advantage of timeseries data analysis techniques developed for fMRI, including functional connectivity and multivariate analysis approaches.