Brightening the Study of Listening Effort with Functional Near-Infrared Spectroscopy: A Scoping Review

Abstract

Listening effort is a long-standing area of interest in auditory cognitive neuroscience. Prior research has used multiple techniques to shed light on the neurophysiological mechanisms underlying listening during challenging conditions. Functional near-infrared spectroscopy (fNIRS) is growing in popularity as a tool for cognitive neuroscience research, and its recent advances offer many potential advantages over other neuroimaging modalities for research related to listening effort. This review introduces the basic science of fNIRS and its uses for auditory cognitive neuroscience. We also discuss its application in recently published studies on listening effort and consider future opportunities for studying effortful listening with fNIRS. After reading this article, the learner will know how fNIRS works and summarize its uses for listening effort research. The learner will also be able to apply this knowledge toward generation of future research in this area.

Speech perception involves the extraction of meaning from a continuous stream of sound. This process appears deceptively simple until it starts to falter, and it varies in difficulty with respect to factors that are both external and internal to the listener. ¹ Real-world communication often involves listening to speech in challenging conditions. These challenges may arise from environmental sources (e.g., background noise, competing talkers), degraded or difficult speech (e.g., conversation via phone, a foreign-accented talker, speech in the listener's nonnative language), or internal factors such as hearing loss or disordered auditory processing. Decades of research have been directed at understanding how listeners perceive speech under a variety of challenging conditions, with common techniques involving behavioral and/or physiological measures of listening effort. Behavioral approaches include self-report and experimental measures, such as dual-task procedures and memory paradigms. Physiological approaches can be divided into measures of the autonomic nervous system (ANS), such as skin conductance or pupil dilation, and neurophysiological measures such as those which can be obtained with functional magnetic resonance imaging (fMRI), electroencephalography (EEG), or magnetoencephalography (MEG; see Table 1 for abbreviations used throughout this article). ² Neurophysiological measures are likely to be the most reliable method of assessing listening effort, as they directly index brain activity associated with increased cognitive effort. A systematic review by Ohlenforst et al ³ concluded that only studies with brain-based measures of listening effort were able to demonstrate greater listening effort in individuals with hearing loss compared to those with normal hearing, suggesting that behavioral and autonomic measures are unable to reliably identify differences in listening effort based on hearing status. However, the most commonly used neuroimaging techniques (e.g., fMRI, MEG/EEG) each pose challenges for obtaining measures of listening effort from hard-of-hearing or deaf individuals who use hearing instruments, such as hearing aids or cochlear implants. In the case of fMRI and MEG, the metallic components in hearing instruments amount to an exclusion criterion in the majority of cases. In addition, motion artifacts are highly problematic in fMRI research, which leads to paradigms that necessitate severe restriction of movement. In the case of EEG, hearing instruments generate an electromagnetic field which leads to artifacts that can be difficult to fully remove.

Table 1. Important abbreviations.

ANS	Autonomic nervous system
DLPFC	Dorsolateral prefrontal cortex
EEG	Electroencephalography
fMRI	Functional magnetic resonance imaging
fNIRS	Functional near-infrared spectroscopy
HbO	Oxygenated hemoglobin
HbR	Deoxygenated hemoglobin
HbT	Total hemoglobin
HD-DOT	High-density diffuse optical tomography
IFG	Inferior frontal gyrus
IPS	Intraparietal sulcus
IPL	Inferior parietal lobule
L1	First (native) language
L2	Second language
MEG	Magnetoencephalography
MFG	Middle frontal gyrus
MTG	Middle temporal gyrus
PET	Positron emission tomography
PFC	Prefrontal cortex
SIN	Speech-in-noise
SNR	Signal-to-noise ratio
STG	Superior temporal gyrus
VAE	Virtual acoustic environment

Recently, functional near-infrared spectroscopy (fNIRS) has increased in popularity for use in cognitive neuroscience research, and it has great potential for overcoming some of the challenges faced with other brain-based measures of listening effort. fNIRS is a noninvasive technique which directs near-infrared (NIR) light into the brain to assess changes in oxygenated and deoxygenated blood flow to cortical brain tissue over time. It has a higher temporal resolution than fMRI, and higher spatial resolution than EEG. It is also compatible with hearing instruments, robust to motion artifacts, and is able to image several areas of cortex that have been implicated in listening effort. All of these features suggest that fNIRS has great potential for studying the neural underpinnings of listening effort in diverse populations and in a variety of ecological contexts. The goals of the current article are to (1) explain why fNIRS is poised to play an essential role in listening effort research, (2) survey the work that has been completed thus far using fNIRS to study listening effort, and (3) identify areas for future research that would benefit from fNIRS within the body of work on listening effort.

Previous Work in Listening Effort

A consensus paper by Pichora-Fuller and colleagues ⁴ established a framework that defined listening effort as “the deliberate allocation of mental resources to overcome obstacles in goal pursuit when carrying out a [listening] task.” By this definition, effortful listening involves three primary elements: task demands, hearing difficulties, and the listener's level of motivation to understand what they are hearing. Many different types of tasks and measures of listening effort have been used and compared in previous research, which can be broadly categorized into behavioral and physiological measures. Behavioral tasks include both subjective and experimental measures. In subjective measures, participants self-report the level of effort they feel they are expending to complete a challenging listening task. Commonly-used self-report measures include standardized and validated scales like the NASA-TLX, ^{5 6} or simpler indicators such as rating effort on a multi-point Likert scale. ^{7 8} Experimental paradigms largely quantify listening effort by accuracy and/or reaction time: As listening challenges increase, accuracy typically decreases, and reaction time increases. Common experimental methods to assess listening effort include dual-task paradigms, in which participants complete a speech recognition task simultaneously with a secondary task involving other cognitive processes (e.g., visual perception or memory). ^{9 10 11} An underlying assumption in dual-task procedures is that domain-general resources that are used to support listening effort should be less available to support secondary tasks that are unrelated to speech understanding. Hence, we should be able to assess listening effort by way of performance on the secondary task. Memory paradigms are also often used on their own to assess listening effort, with the reasoning that as listening challenges increase, fewer domain-general resources are available for memory processing, therefore leading to lower accuracy and slower reaction times in the memory task. Listening span tasks, ^{7 12} the cognitive spare capacity test, ^{13 14} and running memory tasks ¹⁵ are some common memory-based paradigms for behavioral assessments of listening effort.

Behavioral tasks can often be used simultaneously or in addition to physiological measures of listening effort, which involve recording physiological responses from the ANS or the brain. The sympathetic nervous system branch of the ANS responds to listening challenges with a host of physiological changes. Pupillometry has become a standard methodology in the assessment of listening effort, as changes in pupil dilation tend to predict changes in cognitive effort across several tasks and sensory domains. ^{16 17 18} Pupil dilation increases with reductions in the signal-to-noise ratio (SNR) of speech in noise, ¹⁹ greater degradation of speech via noise vocoding, ²⁰ and with increasing syntactic complexity of speech, ²¹ suggesting that it is sensitive to a variety of listening challenges. Another ANS-based measure of listening effort is galvanic skin response, as increased skin conductance has been found with increased difficulty of listening conditions. ^{6 22} Finally, heart-related measures have been found to correlate with increased listening challenges, including heart rate variability, ^{6 23} preejection period, ²⁴ and blood volume pulse amplitude. ²⁵ Elevated cortisol levels have also been linked to effortful listening—however, the endocrine response appears to be physiologically distinct from ANS measures, reflecting changes in stress that persist over time (i.e., chronic) due to hearing challenges. ^{26 27} Consistent with this view, momentary elevations in cortisol have not been found in relationship to situational factors that affect listening challenges.

Brain-based measures of listening effort have typically involved use of fMRI or MEG/EEG. Each neuroimaging technique has highlighted neural signatures related to effortful listening. In the fMRI literature, processing of easily intelligible speech generally involves domain-specific regions including bilateral temporal regions including superior temporal gyrus (STG) and middle temporal gyrus (MTG), as well as inferior frontal gyrus (IFG). ^{28 29 30} As listening challenge increases, domain-general networks are recruited to process the target signal (for a more detailed review, see Peelle, 2018). ³¹ The cingulo-opercular (or “salience”) network, including the anterior insula, cingulate cortex, and thalamus, may become active as part of a general compensatory response leading to the intrinsic maintenance of tonic alertness. ^{32 33 34} In some circumstances, a listening challenge may preferentially engage the frontoparietal attention (or “central executive”) network which directs attention in favor of relevant information while suppressing irrelevant information. ^{35 36} This network runs dorsally from frontal to parietal cortex, including the dorsolateral prefrontal cortex (DLPFC), intraparietal sulcus (IPS), and to the inferior parietal lobule (IPL). Listening effort work with EEG and MEG has also identified oscillatory signatures and auditory-evoked potentials that correlate with listening challenge. Notably, increases in the power of alpha oscillations have been linked to the acoustic degradation of a signal, and therefore may reflect increased engagement of cognitive resources required to suppress noise such as selective attention or working memory. ^{37 38 39} This alpha-modulated noise suppression has been localized to both frontal and parietal sources, ^{40 41} which is consistent with activations observed in fMRI studies of effortful listening. Slower cortical oscillations in the delta and theta range have also been observed to phase-lock with the speech envelope. ⁴² This phase-locking increases when speech is degraded or presented in noise, suggesting modulation of the cortical response to challenging speech by top-down cognitive functions. ^{43 44 45} In terms of event-related potentials, a larger and slower auditory-evoked N1 has been linked to more challenging listening conditions, ⁴⁶ and the N400 amplitude has been linked to linguistic challenge when listening to nonnative accented speech. ⁴⁷

Some of these neuroimaging techniques have also been used successfully to study listening effort in individuals with hearing loss and hearing instruments (see Kuchinsky & Vaden, 2020 for more detail). ⁴⁸ However, as noted earlier, there are major challenges with fMRI and MEG/EEG when trying to study listening effort in populations with hearing instruments. Hearing aids and cochlear implants are incompatible for use in an fMRI scanner due to the metallic components in these devices, which also leave electromagnetic artifacts in recording from most MEG systems. While hearing aids and cochlear implants can be worn during EEG recording, they also inevitably result in electrical artifacts. Some data cleaning methods (e.g., independent components analysis) have been developed to reduce these artifacts, though they cannot remove artifacts completely without introducing some distortion of the signal. It is also possible to use insert earphones that are calibrated against probe microphone measurements taken in situ with hearing aids. ^{49 50} This approach avoids electrical artifacts, but it does not generalize to dynamic contexts outside the lab where sound cannot be fully controlled, and even in the case of a lab study, the realism of such simulations can be called into question because of possible effects related to differences in occlusion between real-life listening with hearing aids and laboratory simulations. These difficulties of standard neuroimaging systems complicate the ability to study listening effort in hard-of-hearing populations who use hearing instruments, and to determine how use of a hearing instrument may affect listening effort.

Introduction to Functional Near-Infrared Spectroscopy

fNIRS has a relatively recent history for use in cognitive neuroscience research. It has many benefits for auditory research and great potential for future work understanding listening effort, especially with regard to testing in ecological contexts and with those with hearing loss who use hearing instruments. fNIRS is a form of optical imaging, as it measures the hemodynamic response function of the brain with the use of NIR light (see Pinti et al, 2020, for complete review). ⁵¹ Fig. 1 illustrates the mechanics of the fNIRS signal. NIR light generated from low-power lasers and/or LEDs is shone into the head from a light source placed on the scalp. This light scatters through the scalp, skull, cerebrospinal fluid, and other layers of tissue. As the body is primarily composed of water, it minimally absorbs the NIR light, and the light is therefore able to continue traveling through the tissue. ⁵² Comparatively, hemoglobin is the most absorptive chromophore within the NIR optical window, so it absorbs more of the NIR light relative to surrounding tissue and cerebrospinal fluid. Two wavelengths of NIR light are typically used to measure absorption for oxygenated and deoxygenated hemoglobin. NIR light at 700 nM is “redder” and tends to be better absorbed by deoxygenated hemoglobin (HbR), while NIR light at 900 nM is less visible and is better absorbed by oxygenated hemoglobin (HbO). ⁵³ A light detector (also called a “target”) is placed a specific distance away from the light source on the scalp, and it measures the optical density of backscattered light at both wavelengths that has not been absorbed.

Figure 1.

Illustration of the mechanics of the functional near-infrared spectroscopy signal. Near-infrared photons are emitted from the light source and travel through the layers of the head to the detectors. The channel indicating the measurement point on the cortex is indicated with C . The depth of the channel is proportional to the distance between the source and detector on the scalp ( C indicates a standard-depth channel; SSC indicates a shallower short-separation channel).

A source–detector pair creates a channel, which is located at the midpoint between the source and the detector. The area of cortex being imaged can be found perpendicular to the scalp beneath the midpoint at a depth corresponding to half of the distance between the source and detector pair. When the source and detector are farther apart, the detector is able to measure backscattered light at a greater depth than when they are closer together. However, if the distance between the source and detector pair is too close, light will reach the detector directly without traveling through the scalp. If the distance is too far, the detector is unable to detect any meaningful amount of light. Research suggests that to reach the cortex, the optimal distance between a source and a detector is approximately 3 to 5 cm in adults and 2 to 3 cm in infants. ⁵⁴ Some fNIRS systems also include “short separation channels,” which are source–detector pairs placed at a closer distance (approximately 1–1.5 cm) so that the channel measures at a shallower depth than other source–detector pairs. These short channels enable tracing of physiological processes that are not indicative of neural activity, such as respiration or changes in noncortical blood flow that result from ANS activity. The activity captured by short channels can then be regressed out of the fNIRS recording during analysis as a means of removing physiological noise from standard channels, thus providing a cleaner picture of the hemodynamic activity related to local cortical circuits. During analysis, optical density measurements captured by the detectors over time are converted into hemoglobin concentrations that correspond to the two different wavelengths of NIR light emitted by the sources. HbO and HbR concentrations may be analyzed separately, or HbR and HbO may be added together to analyze the total changes in hemoglobin concentrations (HbT). The majority of studies focus on analyses related to hypothesis testing on HbO, while reporting descriptive statistics on HbO as well as HbR. Some studies have used the difference between HbO and HbR as a measure for analysis due to its relative sensitivity to both the increase in HbO and the decrease in HbR. ^{55 56}

These analyses of oxygenation and deoxygenation often stand on their own, but the same measures may also be subjected to connectivity analysis in order to better understand network activity. These connectivity analyses can be conducted with measures obtained from different regions of interest while the participant is in a resting state, often for the purposes of understanding individual differences or neuroplasticity. ⁵⁷ Alternatively, the analyses may be performed on measures collected while the participant is engaged in a task, often for the purposes of understanding differences in circuitry underlying different tasks. ⁵⁸ The assessment of connectivity can be as simple as correlating two time series from different regions of interest, but may also take on more complex operations in the form of wavelet coherence analysis. ⁵⁹

While the use of optical imaging for cognitive neuroscience is fairly recent, fNIRS has been in development for neuroimaging purposes since the early 1990s. At first, single-channel devices were developed and could detect hemodynamic changes in the brain in response to basic functional tasks. ^{60 61 62 63} Eventually, several single-channel devices were combined and used simultaneously to measure from multiple regions at once ⁶⁴ before the development of multichannel systems (see Ferrari & Quaresima, 2012, for more complete history). ^{65 66 67}

Most commercially available systems for optical imaging of the brain are continuous wave (CW) instruments, which continuously emit NIR light at 2 to 3 wavelengths for measurement of HbO and HbR (see Scholkmann et al, 2014, for detailed review). ⁶⁸ However, time-domain and frequency-domain devices can also be used to collect optical data. Frequency-domain devices produce intensity-modulated NIR light, while time-domain devices emit rapid pulses of NIR light. Current fNIRS systems are either fiber-based or wearable devices and generally involve attaching the optodes to a cap, band, or other headgear. Fiber-based systems use optical fibers containing low-power lasers, so the participant remains stationary during recording and mobility is limited. However, optical fibers have higher power than LED light and can image the cortex with greater separation between the source and detector pairs (up to 6 cm), thereby allowing deeper penetration of the cortex. Wearable systems do not use fibers, and instead employ LEDs to emit the NIR light into the scalp. Because wearable systems are fiber-less, they are more portable, and can be used to record from a participant in motion ⁶⁹ and even outside the lab. ⁷⁰ Both types of systems have varying degrees of optode density and can generally accommodate up to approximately 64 channels. Recently, high-density diffuse optical tomography (HD-DOT) systems that are both fiber-based and LED-based have been developed to allow measurement with NIR light from a larger number of optodes (more than 90 sources and detectors). These systems are designed so that sources and detectors are placed at several different separation distances, which allows for recording from spatially overlapped channels and brings the spatial resolution of fNIRS approximately equal to that of 3-T fMRI. ^{71 72}

Several features of fNIRS make it a particularly good choice for studies in auditory cognitive neuroscience (see Table 2 ). The system itself is generally quiet, so it is ideal for auditory tasks where spatial imaging is desired. The fMRI scanner produces noise during imaging, though some techniques have been developed to facilitate presentation of auditory stimuli in the scanner. ⁷³ fNIRS systems vary in the amount of noise they emit: Most fNIRS systems are either silent or generate low levels of ambient noise that are far less disruptive than fMRI scanner noise. One example in which imaging noise may have affected results is in the study of auditory working memory. Rovetti et al (2021) ⁷⁴ found that there was no difference in DLPFC activity for auditory versus visual n-back tasks while recording with a silent fNIRS system, which was consistent with findings from a previous positron emission tomography (PET) study ⁷⁵ but contrary to prior fMRI results suggesting increased DLPFC engagement for auditory relative to visual n-back tasks. ^{76 77} One potential explanation for this pattern of findings across studies is that the fMRI scanner noise contributed to additional listening effort, thus increasing engagement of DLPFC for the auditory task relative to the visual task. PET and fNIRS are both very quiet by comparison, and therefore would not suffer from the same confound. As previously mentioned, fNIRS is also compatible with hearing instruments such as hearing aids and cochlear implants. There is no interference from the metallic components of the devices (which prohibits these devices from being used in an MRI scanner), and as fNIRS measures hemodynamic activity, the signal is not affected by the electrical components of hearing devices.

Table 2. Comparison of popular neuroimaging methods.

Feature	fMRI	EEG	fNIRS
Acoustic noise level during imaging	High	Silent	Silent-low
Compatible with hearing instruments	No	Yes a	Yes
Spatial resolution	Good	Poor	Fair
Temporal resolution	Poor	Good	Fair
Portability	No	Few systems	Some systems
Robust to motion artifacts	No	No	Yes

^aWhile hearing aids and cochlear implants can be worn during EEG recording, they produce an electromagnetic artifact.

The separate measurement of HbO and HbR in fNIRS can be viewed as another advantage over fMRI. HbR measured with fNIRS correlates most strongly with the blood oxygen level–dependent signal (BOLD) measured with fMRI. ⁷⁸ For example, in a concurrent fMRI-fNIRS study, Huppert et al (2006) ⁷⁹ found correlation coefficients between 1/BOLD and HbT, HbO, and HbR to be 0.53, 0.71, and 0.98, respectively. To explain the discrepancy in correlation strengths, we can consider biophysical models of hemodynamics. ⁸⁰ When a population of neurons becomes active, HbO is expected to rush into the area. As a consequence of this increase in HbO, HbT will temporarily increase and HbR will be washed out. Thus, over the course of an entire block of trials, HbO and HbR will be inversely correlated but over the short term (e.g., the first 10 seconds following the onset of a stimulus), HbO and HbR may differ to some extent. The ability to measure HbO directly, and to assess HbO and HbR independently, affords fNIRS a more nuanced temporal view of brain hemodynamics than fMRI. This nuanced temporal view is further supported by the relatively high sampling rate of fNIRS (10–100 Hz) compared to fMRI (typically 0.5 Hz), which facilitates use of event-related, block, and mixed designs. ⁸¹

fNIRS has several additional advantages over other imaging systems that may make it especially suitable for listening effort research. It has higher spatial resolution than EEG (10–20 mm for fNIRS compared to 5–9 cm for EEG), so it is a comparably better choice for localizing cortical activity. ⁸² The locations of source and detectors on the scalp can also be digitized and converted to coordinates in MNI space and/or coregistered to fMRI brain atlases for better visualizations of localized activity. Certain fNIRS systems can also interface with other imaging modalities to provide a more thorough, multimodal assessment of neural activity. Systems have been developed for recording from concurrent EEG-fNIRS, ^{83 84} fMRI-fNIRS, ^{85 86 87} and MEG-fNIRS. ⁸⁸ The cost of the average fNIRS system is comparable to EEG, and therefore relatively low when compared to fMRI, MEG, or PET systems. ⁵¹ fNIRS systems also do not have specific environmental requirements, such as magnetically or electrically shielded rooms. They can be used for longer imaging sessions up to approximately 1 hour and can be easily implemented for repeated measures designs. ⁸² An additional benefit for fNIRS systems is that they are relatively robust to motion artifacts, so they can be used in paradigms that involve substantial participant movement. ⁷⁰ Wearable fNIRS systems are portable and can be used in more naturalistic settings outside the lab. Overall, the many benefits of fNIRS systems indicate great potential for brain-based research.

fNIRS systems are not without their limitations. As noted, the penetration depth of fNIRS channels is only about 1.5 to 3.5 cm from the scalp in adults, depending on the distance between source–detector pairs. This means that fNIRS cannot measure activity from deeper cortical and subcortical structures. This poses a challenge for imaging the auditory cortex, as it has a transverse orientation within the temporal lobe, and the NIR light cannot reach the entire auditory cortex. ⁸⁹ Nevertheless, attempts to image parts of the auditory cortex have been successful, including the primary auditory cortex, secondary auditory cortex, and Wernicke's area. Fig. 2 depicts the locations of these regions of interest, in addition to Broca's area and core nodes of the frontoparietal attention network.

Figure 2.

Model representation of cortical regions of interest for listening effort, including left temporal, frontal, and parietal sources.

Additionally, NIR light transmission is attenuated by darker hair and skin pigmentation, which has higher absorption in the NIR optical window. ^{90 91} Neuroimaging applications of fNIRS therefore risk limited racial and ethnic diversity because a reliable signal may not be attainable from individuals with darker hair and skin tones. However, the extent of this problem appears to be negligible in recent fNIRS systems and may be mitigated with the inclusion of short channels. ^{92 93}

Finally, while some laser- and LED-based fNIRS systems are silent, others generate noticeable levels of noise that could be distracting or serve as confound in auditory experiments. The noise level tends to correlate with the size of the system; so, wearable/portable systems are more likely to be silent while fiber-based systems are likely to generate more low-level noise from the cooling fans and other components inside the amplifier box. If using a fiber-based system for research involving auditory stimuli (e.g., listening effort studies), it is recommended to place the fNIRS amplifier outside the sound-attenuated booth to avoid this confound.

fNIRS and Auditory Cognitive Neuroscience

The earliest study to successfully image auditory cortex with fNIRS was completed by Hoshi and Tamura, ⁶⁰ who combined five single-channel devices placed over different areas of the cortex and found systematic increases in activation to sound in the temporal lobe. Ohnishi et al ⁹⁴ followed up on this work to more specifically measure from auditory cortex. They first used MEG to localize the auditory cortex response to a pure tone, then placed an fNIRS channel over this region and confirmed an increase in both HbO and HbR in response to the tone stimulus. Further work extended use of fNIRS to capture the hemodynamic response to sound in auditory cortex in children and infants. ^{95 96 97} Differential activations to various sound categories in auditory cortex have also been confirmed using fNIRS, ^{98 99} indicating that while fNIRS has lower spatial resolution than fMRI, it can be quite sensitive to changes in the type of sound stimulus being used. To our knowledge, no study to date has confirmed whether fNIRS has the necessary sensitivity for tonotopic mapping of the auditory cortex.

Once early research confirmed that fNIRS can capture the hemodynamic response to acoustic stimuli, it became a more popular tool in several areas of research within auditory cognitive neuroscience. One of its most prominent uses is to study auditory processing in deaf and hard-of-hearing individuals, including those who use hearing instruments (see Oghalai, 2017; Bortfeld, 2019, for detailed review). ^{100 101} As previously mentioned, fNIRS is uniquely compatible with hearing aids and cochlear implants, and is therefore an ideal choice for noninvasive neuroimaging in individuals who use hearing devices. Furthermore, fNIRS is simple to implement over multiple testing sessions; so, it is ideal for longitudinal designs and may be an important tool to assess the efficacy of hearing devices over time (e.g., before and after cochlear implantation). ¹⁰² Sevy et al ¹⁰³ were the first group to identify a reliable response to speech in auditory cortex for cochlear implant users. Bisconti et al ¹⁰⁴ found overall similar activation patterns in auditory and nonauditory regions between cochlear implant users and normal-hearing adults during passive listening to speech, suggesting that hearing loss in adulthood does not significantly alter the neural organization of speech perception. Cortical activation to speech in cochlear implant users has also been associated with speech perception accuracy, which links neural variability to behavioral variability in cochlear implant outcomes. ^{105 106} More recently, HD-DOT has been explored as a means to increase spatial resolution of imaging in cochlear implant users. ¹⁰⁷ In contrast, relatively few studies have investigated auditory perception in hearing aid users with fNIRS. Two studies to be discussed later tested adult and children who use hearing aids in listening effort tasks, but other fNIRS work in hearing aid users is currently limited. ^{108 109}

Some studies have investigated cross-modal plasticity in cochlear implant users by measuring the cortical response to auditory and visual stimuli, as prior fNIRS evidence suggests that neural reorganization occurs during deafness to maximize processing of visual and other sensory stimuli in absence of auditory stimulation. ^{110 111} Chen et al completed concurrent EEG-fNIRS recording during presentation of unimodal and multimodal stimuli, finding greater visual adaptation and lower activation of visual cortex in cochlear implant users relative to normal-hearing adults. ^{112 113} Anderson et al ¹¹⁴ measured oxygenation of superior temporal cortex in cochlear implant recipients in response to visual speech stimuli both preimplantation and 6 months postimplantation, finding that an increased response to visual speech in auditory cortex postimplantation was associated with higher speech perception scores with a cochlear implant, therefore suggesting that increased cross-modal plasticity in auditory cortex has adaptive benefits for speech perception postimplantation. Follow-up work by the same group showed that fNIRS can be used preimplantation to predict cochlear implant outcomes in deaf adults, as stronger auditory cortex activation to visual speech preimplantation was associated with poorer auditory speech perception 6 months postimplantation. ¹¹⁵ They also investigated cross-modal activations to speech in pediatric cochlear implant recipients, finding that cochlear-implanted children had greater auditory cortex responses to visual speech but similar responses to auditory speech compared to normal-hearing children. ¹¹⁶

In normal-hearing listeners, fNIRS has been used successfully with a variety of speech stimuli to replicate and extend findings from other neuroimaging methods. Differential cortical activations were identified for normal speech versus speech that was distorted to simulate the signal transmitted via a cochlear implant, confirming that fNIRS can detect differences in the cortical response to speech based on intelligibility. ¹¹⁷ In infants, differential activations were found for listening to speech in the infants' native versus nonnative language at 1-day-old ¹¹⁸ and for native-accented versus foreign-accented speech within the first few days of birth. ¹¹⁹ Audiovisual speech stimuli have also been shown to increase activity in temporal regions relative to auditory-only stimuli in adults ¹²⁰ as well as infants. ¹²¹ Shader et al ¹²² identified differential activation in specific regions of interest that responded to auditory speech near the auditory core, and visual speech near middle occipital gyrus.

fNIRS has also been used to understand the neural processing of complex sound beyond the types of stimuli that are used in typical speech perception studies. For example, differential cortical activations have been observed for music versus noise, ¹²³ various genres of music, ¹²⁴ familiarity of music, ¹²⁵ and for whether or not music is playing during memory encoding. ^{126 127 128} Differential responses have also been observed across categories of vocal emotion in adults ^{129 130 131} as well as infants. ¹³² Given the portable nature of many systems, fNIRS also has great potential for studying dynamic communication in real-world environments via a process called hyperscanning. ^{133 134 135}

fNIRS and Listening Effort

Listening effort is a burgeoning area of hearing research with important implications for real-world scenarios that involve auditory–cognitive interactions. The neural substrates of these interactions can be effectively studied using fNIRS. Researchers are able to take advantage of the benefits of optical imaging for auditory research, as well as the flexibility and ease of use of fNIRS systems across diverse populations, including people living with hearing loss who use hearing instruments. Because of its versatility, the use of fNIRS to study listening effort spans several disciplines, including psychology, neuroscience, audiology, linguistics, and cognitive science. An online search was conducted in February 2022 to locate fNIRS studies of listening effort using Google Scholar, as the diversity of disciplines potentially involved in this work is unlikely to be fully captured using a single-research database. The search included the following search terms: (listening effort) and (fNIRS or functional near infrared spectroscopy). After limiting the search results to peer-reviewed original research articles, 61 studies were identified. Next, the article titles and abstracts were reviewed to determine whether each article used fNIRS to study listening effort. On the basis of this assessment, 42 of the 61 articles were eliminated. The authors then read the method and results sections of the remaining 19 articles to further determine eligibility, and eliminated all articles that did not have listening effort as the primary focus of the research. A replication of the same search process conducted using PubMed and PsycInfo yielded no new studies that were not already indexed by Google Scholar.

Table 3 lists the 11 currently published original research articles as of February 2022 that have used fNIRS to study listening effort. These studies have all been published within the past 5 years, which is an indication of the relatively recent adoption of fNIRS in listening effort research. A variety of listening challenges and stimulus manipulations have been used, allowing for comparison across studies of different sources of listening challenge and the accompanying effects on neural processing.

Table 3. Currently published fNIRS studies of listening effort.

Citation	Subject population	fNIRS system	Cortical ROIs	fNIRS signal analyzed	Listening challenge	Stimulus type
Bell, Peng, Pausch et al (2020)	Children with normal hearing and hearing aids	Hitachi ETG-4000 system with 44 channels	Bilateral STG and IFG	HbT	Acoustic	Speech-in-noise (virtual acoustic environment)
Defenderfer, Forbes, Wijeakumar, Hedrick, Plyler, & Buss (2021)	Young adults with normal hearing	Techen CW7 system with 13 channels, 1 short channel	Left frontal and temporal cortices	HbO; HbR	Acoustic	Speech-in-noise; degraded speech
Defenderfer, Kerr-German, Hedrick, & Buss (2017)	Young adults with normal hearing	System not stated; 12 channels	Bilateral auditory cortex, STG	HbO; HbR	Acoustic	Speech-in-noise, degraded speech
Lawrence, Wiggins, Anderson, Davies-Thompson, & Hartley (2018)	Young adults with normal hearing	Hitachi ETG-4000 system with 44 channels	Bilateral superior temporal cortex, IFG, IPL	HbO; HbR	Acoustic	Degraded speech
Lawrence, Wiggins, Hodgson, & Hartley (2021)	Children with normal hearing	Hitachi ETG-4000 system with 26 channels	Bilateral superior temporal cortex, IFG, rolandic operculum, posterior temporal cortex	HbO	Acoustic	Degraded speech
Mushtaq, Wiggins, Kitterick, Anderson, & Hartley (2021)	Children with normal hearing	Hitachi ETG-4000 system with 44 channels	Bilateral superior temporal cortex, IFG, rolandic operculum, posterior temporal cortex	HbO; HbR	Acoustic	Degraded speech
Rovetti, Goy, Pichora-Fuller, & Russo (2019)	Older adults with normal hearing and hearing aids	fNIR Imager 1100 with 16 channels	Bilateral prefrontal cortex	HbO–HbR	Acoustic	Visual and auditory verbal n-back tasks, completed with/without hearing aids
Rovetti, Goy, Zara, & Russo (2021)	Young adults with normal hearing	fNIR Imager 1100 with 16 channels	Bilateral prefrontal cortex	HbO–HbR	Acoustic; linguistic	Speech-in-noise, sentences with high/low context
Rowland, Hartley, & Wiggins (2018)	Young adults with normal hearing	Hitachi ETG-4000 system with 52 channels	bilateral prefrontal cortex, superior temporal cortex	HbO; HbR	Acoustic	speech-in-noise (acoustic scenes)
White & Langdon (2021)	Young adults with normal hearing	NIRx NIRScout system with 50 channels	Bilateral IFG, SMG, primary somatosensory cortex, MFG	HbR	Acoustic; linguistic	Sentences varying in semantic plausibility, syntactic complexity, and speech rate; degraded speech
Wijayasiri, Hartley, & Wiggins (2017)	Young adults with normal hearing	Hitachi ETG-4000 system with 52 channels	Bilateral prefrontal cortex, IFG	HbO; HbR	Acoustic	Degraded speech

Acoustic Challenges to Listening

The majority of studies manipulated listening effort with some manipulation of acoustic challenge—most commonly, presenting one or more varieties of speech in noise or speech degradation. Wijayasiri et al ¹³⁶ were one of the first groups to investigate potential neural correlates of effortful listening with fNIRS using degraded speech. Young adults with normal hearing were presented simultaneously with a sentence and a nonspeech auditory distractor. The sentence was either presented unaltered (clear) or degraded by noise-vocoding, which is created by splitting the speech signal into frequency bands, then extracting the amplitude envelope and using that envelope to modulate noise in that frequency band. The bands are then recombined to create the full noise-vocoded signal. ¹³⁷ The authors used four channels in their degraded speech, which was intended to keep the speech intelligible while reducing spectral clarity. Participants were instructed to attend to the speech or the distractor. They found that left IFG activation increased for degraded relative to clear speech, but only when attention was directed toward the speech—when attention was directed toward the distractor, there was no difference in IFG activation for degraded versus clear speech. Additionally, when listening was directed to degraded speech, the peak in the hemodynamic response was delayed in the left IFG relative to the superior temporal cortex. These results aligned with prior fMRI work suggesting that understanding-degraded speech relies on attention. ¹³⁸

Lawrence et al ¹³⁹ expanded on the use of degraded speech in effortful listening—instead of a single level of degradation, they developed five levels of intelligibility for eight-channel noise-vocoded speech by raising the extracted acoustic envelope of each channel to a fractional power, thereby manipulating the depth of envelope modulation for each channel. Behavioral pilot testing allowed the fractional power of each condition to be set so that there was a target accuracy of 0, 25, 50, 75, and 100% correct keywords. Participants listened to the vocoded Bamford–Kowal–Bench (BKB) sentences at the five levels of intelligibility and indicated whether a probe word that appeared on the screen had been spoken in the sentence they heard on each trial. They found that activity in bilateral superior temporal cortex increased monotonically with speech intelligibility, suggesting that as speech becomes more intelligible (i.e., less effortful to understand), engagement of auditory regions increases. In left IFG, however, activation was highest for the 25 and 50% intelligibility conditions, with the lowest activation for the most challenging 0% intelligible condition. This seems to suggest that when speech is impossible to understand, the listener may not attend to the signal, therefore decreasing engagement of left IFG relative to circumstances when speech is challenging but still intelligible. Some channels over bilateral inferior parietal regions (IPS and IPL) and middle temporal lobe also showed deactivation that was most pronounced for intermediate levels of intelligibility, indicating a “ U -shaped” pattern of deactivation relative to silence where activity in those regions was lower for moderately intelligible speech compared to highly intelligible or unintelligible speech. The authors suggested that deactivation in these more posterior regions reflects attentional demands of effortful listening to moderately intelligible speech. Importantly, greater activation was found in left IFG and bilateral posterior superior temporal cortex for correct versus incorrect trials. This increased activation on correct trials might have been due to increased attention or comprehension.

Two recent studies from the same group extended their investigation of cortical activations to degraded speech to children with normal hearing. Lawrence et al ¹⁴⁰ presented the same degraded speech stimuli from their 2018 study, in addition to a clear speech condition, to children between the ages of 6 and 13. While the children were slower to respond and less accurate overall relative to adults on the behavioral task, they displayed similarly monotonic increases in accuracy as speech intelligibility increased. A similar pattern of increases in activation as speech intelligibility increased from their 2018 study with adults was also observed in left superior temporal cortex in children, but was not found in right superior temporal cortex. This increased left lateralization of speech processing in children has been found in previous pediatric studies using both fMRI and fNIRS and may be related to greater brain–scalp distance in right frontal and temporal regions relative to the left hemisphere, which is less apparent in adults. Activation of left IFG also increased monotonically with increased intelligibility, which was a departure from the adult findings that left IFG activation was greatest for intermediate levels of intelligibility and lower for highly intelligible and unintelligible speech. The authors suggest that some listening effort was required even when speech was most intelligible, as children did not reach 100% accuracy in the task. In addition, while a trend toward similar patterns of activation for correct versus incorrect trials was found in children who approximated the adult findings from 2018, the result was not statistically significant after correcting for multiple comparisons.

Mushtaq et al ¹⁴¹ used the same BKB sentence stimuli with children aged 6 to 12, but with an additional method of signal degradation. Instead of all conditions using 8-channel noise-vocoded speech, they utilized three levels of vocoding with one, four, and eight channels, with one channel being least intelligible and eight channels being most intelligible. The eight-channel speech was also manipulated with full, reduced, and zero modulation depth of the amplitude envelope in a manner similar to Lawrence et al, ^{139 140} with resulting intelligibility decreases as modulation depth was reduced. They again found that activation in left temporal regions increased as speech intelligibility increased, as in their 2018 study in children, while deactivation relative to silence in right inferior parietal regions increased with greater intelligibility. Unlike the findings of the previous two studies, however, there was no significant relationship between intelligibility and activation of left IFG, which the authors suggest may be due to insufficient spatial resolution in this region of interest.

An additional source of acoustic challenge for listening may arise from the background—the speech signal itself remains intact, but background noise increases the effort required to understand the speech. Speech-in-noise (SIN) is a common manipulation of listening effort, and several fNIRS studies have used various kinds of noise to assess the cortical mechanisms associated with this type of effortful listening. Defenderfer et al ¹⁴² completed a direct comparison between SIN and degraded speech by presenting sentences either in quiet, with a −2.8 dB SNR, or with eight-channel noise vocoding to young adults with normal hearing while imaging the left temporal lobe. Participants were asked to repeat back the sentence after each presentation, and behavioral results suggested that they were most accurate for speech in quiet and least accurate for speech in noise. Oxygenation was greatest at several left temporal channels for SIN relative to speech in quiet and noise-vocoded speech, and oxygenation at a subset of channels was also higher for correct versus incorrect trials when collapsed over the three conditions. However, the authors did not find a correlation between individual differences in accuracy and oxygenation at any of the channels. In a follow-up study, Defenderfer et al ¹⁴³ executed a similar design but with changes in the manipulation of intelligibility. They also included speech in quiet and eight-channel noise-vocoded speech as conditions, but used SIN at two different levels of intelligibility (+10 dB and −4 dB), and added a condition that combined SIN and noise vocoding. They also imaged bilateral inferior frontal and temporal cortices, rather than focusing solely on the left temporal cortex. The combination of SIN and vocoding was the most challenging condition, leading to lowest behavioral accuracy on the task. fNIRS channels corresponding to MTG and middle frontal gyrus (MFG), a large patch of the frontal lobe that encompasses the DLPFC, showed increased activation associated with low intelligibility conditions: MTG activity was related to processing the SIN-vocoded speech, while MFG activity increased for SIN. The authors suggested that this differential activation based on noise type indicates that there is less reliance on frontal attention mechanisms for degraded speech that is still intelligible, while frontal engagement for SIN implies an increase in attention to parse the signal out from the noise.

Noise can also reflect more naturalistic listening conditions, thereby increasing the external validity of findings regarding listening effort. Two fNIRS articles have explored using more ecologically valid types of noise to assess listening effort. Rowland et al ¹⁴⁴ used stimuli from the RealSpeech content library, which combines narrative speech with binaural recordings of background sounds from various environments at different SNRs. For the current study, they chose passages with SNRs that varied over time between −16 dB and +25 dB, corresponding to transitions in the acoustic environment (e.g., a swimming pool, cafe, traveling car). The authors used intersubject correlation to analyze differences in fNIRS activation across different environments, which identified bilateral STG, IFG, DLPFC, and frontopolar cortex as regions that responded similarly across participants during attentive listening to speech in complex naturalistic scenes. However, the left IFG and DLPFC did not show the expected statistically significant relationship between levels of activation and the degree of self-reported effort during listening. The authors propose that this finding may be a result of either too much variability in the self-reported effort measure across participants, or the inherent variability in using complex acoustic scenes with higher external validity. Nevertheless, this study was an important step toward studying naturalistic sources of acoustic challenge using fNIRS.

Bell et al ¹⁰⁹ took the ecological validity of stimuli a step further by presenting a complex acoustic scene via a virtual acoustic environment (VAE), which used a four-channel array of speakers at each corner of a sound booth to more closely approximate the acoustic features of sound encountered outside the lab. Bell et al also included an additional source of listening challenge, as half of their participants had hearing loss and used hearing aids. Hearing loss serves as an internal source of acoustic challenge and may compound upon external challenges faced from signal degradation or environmental noise—so it is especially important to understand the additional role of hearing loss/hearing aid use when studying the effort required for real-world listening challenges. This study served as a pilot test of the VAE, using a simulated classroom environment with three children who had normal hearing and three children who used hearing aids. Participants were seated in the booth with a target speaker positioned in front of them, which was emitting a five-keyword target sentence while two-talker distractor speech was presented elsewhere in the room with varying degrees of reverberation to simulate the acoustic properties of difference classrooms. The sound levels of the target speech and environment were calibrated to each child's level of hearing. Participants were asked to attend to and verbally repeat the five-keyword target sentence. Behaviorally, the three children with normal hearing benefitted from lower reverberation and greater spatial separation between the target speaker and distractor speakers. The children with hearing aids had more variable performance, though their performance was worse across all conditions compared to the normal-hearing children. The three children with normal hearing also demonstrated greater activation of left IFG for specific challenging listening conditions (e.g., when the target talker and distractor speech shared the same pitch); however, neural results across hearing-aided children were highly variable and there was no consistent effect in left IFG. Definitive conclusions regarding differential processing of naturalistic SIN for normal hearing versus hearing aids are not possible with such a small sample size, but the VAE shows promise for systematically varying acoustic cues to reflect more ecologically valid listening environments.

Rovetti et al ¹⁰⁸ also investigated cortical correlates of listening effort in hearing aid users. Rather than children with hearing loss, this study tested older adults with hearing loss who use hearing aids. Participants with hearing loss completed a visual verbal n-back task and an auditory verbal n-back task at four levels of working memory demand with and without their hearing aids, comparing them to a control group of adults with normal hearing. The authors specifically targeted frontal regions with their fNIRS configuration, in opposition to other work discussed here that primarily focused on temporal regions with some channels in frontal and/or parietal regions. Participants showed slower reaction times for the highest level of working memory demand (recall of the item presented immediately, 1-back, 2-back, or 3-back), while reaction times were faster for the visual relative to auditory n-back task. Hearing aid use did not affect reaction time, and it did not modulate the relationship between self-reported effort and working memory demand. With respect to brain correlates, oxygenation across the entire prefrontal cortex (PFC) increased as working memory demand increased reaching asymptote at 2-back. Interestingly, there was no difference in PFC oxygenation for the auditory versus visual n-back task, and no effect of hearing aid use on PFC oxygenation. The authors concluded that PFC activation is associated with more challenging listening, as indicated by the increased oxygenation as working memory demands increased in the auditory n-back task. However, the lack of effect of hearing aid use on oxygenation suggests that hearing aids do not necessarily decrease listening effort. One plausible explanation for the lack of an effect of hearing aid on listening effort is that while hearing aids do amplify the target signal, they also amplify other irrelevant background sounds, so they do not reliably increase the SNR of the listening environment. Additionally, they can distort sound or produce feedback, which add further sources of noise.

Linguistic Challenges to Listening

While acoustic challenges tend to be most salient when considering effortful listening, the linguistic aspects of a speech signal can also vary in difficulty and therefore require different amounts of effort for successful understanding. These linguistic challenges range from semantic difficulty to syntactic and lexical complexity, and are inherent to the stimulus itself regardless of acoustic or environmental factors. A study by Rovetti et al ¹⁴⁵ manipulated listening effort via both acoustic and linguistic challenge. They presented normal-hearing young adults with sentences that provided either high or low sentence context. For example, a high-context sentence (“She had to vacuum the rug .”) contains contextual information that aids in predicting the final word of the sentence, while a low-context sentence (“Tom could have thought about the sport .”) is lacking in contextual cues. These high- and low-context sentences were presented in multi-talker babble at two SNRs that varied in acoustic difficulty (+4 dB “easy” vs. −2 dB “hard”). Participants were asked to repeat the last word of the sentence. Similar to their earlier work, ¹⁰⁸ only bilateral PFC was imaged. Behavioral accuracy was lowest for low context/hard SNR speech, and highest for high-context/easy SNR speech. Oxygenation increased across the entire PFC for the hard SNR compared to easy SNR speech, suggesting greater frontal engagement and attention for greater acoustic challenges. There was also a significant effect of context only in the left lateral PFC, with greater oxygenation for low- versus high-context sentences. The left lateral PFC encompasses left IFG as well as left DLPFC. These results suggest that while PFC engages more generally for understanding acoustically challenging speech, left lateral regions of PFC are responsive to both acoustic and linguistic challenges.

White and Langdon ¹⁴⁶ further investigated the interplay between multiple sources of acoustic and linguistic challenges to listening with a plausibility judgment task. Normal-hearing young adults were presented with sentences that were either semantically plausible (e.g., people feeding birds ) or implausible (e.g., birds feeding people ). The sentences were also manipulated in terms of syntactic complexity, with simpler subject-relative clauses (e.g., women that feed birds ) versus more complex object-relative clauses (e.g., birds that women feed ). The acoustic properties of the sentences were manipulated by varying the speech rate (normal vs. fast) and degrading the speech via either eight-channel noise vocoding to simulate cochlear implant speech, or a hearing aid simulation which compressed the speech with a limited spectral range. This resulted in a total of four interacting manipulations of listening challenge: two linguistic and two acoustic in nature. Participants listened to each sentence and made judgments about their plausibility or implausibility, and also self-reported their effort on each trial. Both sources of acoustic challenge (speech degradation and speech rate) predicted task accuracy and reaction time, and self-reported effort correlated with behavioral performance. Additional interactions between speech degradation, speech rate, and syntactic complexity also predicted accuracy. As the degree of listening challenge increased, activity in right MFG increased until the most difficult condition (noise-vocoding × fast speech rate × high syntactic complexity × low plausibility), at which point activation dropped off. The authors suggested that this right MFG activity reflected sustained engagement of auditory attentional resources with increased listening challenge, and the drop in activation for the most difficult condition indicates decreased attention related to a loss in motivation to understand the signal. More generally, these findings are consistent with the proposal that the right MFG is a site of convergence between two neuroanatomically distinct attention networks. ^{147 148} The former is bilateral and thought to be involved with the top-down voluntary allocation of attentional resources, and the latter is largely right-lateralized and thought to be involved with reorientation of attention to unexpected events. ¹⁴⁹

Increased listening challenge also correlated with greater engagement of regions involved in the core speech and language network: easier conditions primarily engaged IFG and STG, and as listening challenge increased, activation in posterior temporal regions and DLPFC increased. The authors noted that this shift in activation suggests a hierarchical organization of linguistic processing, and that these regions parallel the dorsal and ventral streams posed in Hickok and Poeppel's dual stream model of speech perception. ¹⁵⁰ They proposed that easier listening conditions engage more of the ventral stream, and as listening challenges increase, the dorsal stream is recruited to activate associated motor plans that may aid in speech understanding. Note that this dorsal stream of the speech network is primarily left-lateralized and distinct from the bilateral frontoparietal (dorsal) attention network mentioned earlier, though both are engaged more heavily during challenging listening conditions.

Summary and Future Directions of Listening Effort and fNIRS Work

Several of the listening effort fNIRS studies described here have confirmed involvement of regions associated with effortful listening through prior studies conducted with fMRI, EEG, and MEG. Multiple fNIRS studies replicated increased activation of domain-specific resources inclusive of the STG and left IFG in response to greater listening challenges, particularly in regard to increased linguistic challenge. The engagement of right IFG and domain-general frontal areas like the DLPFC also increases with greater listening challenge. Importantly, the two studies that include hearing aid users also provide early evidence of changes in activation in left IFC and DLPFC that correspond to listening challenge, thereby supporting the use of fNIRS to study listening effort in hard-of-hearing populations with hearing instruments. ^{108 109}

There is still much to learn about the interactions between frontal areas associated with motor planning and attention, and temporal regions associated with core auditory processing. Few studies have imaged areas beyond temporal cortices and IFG/DLPFC. Some work with fNIRS has implicated the role of parietal regions as part of a frontoparietal attention network, and additional frontal regions such as anterior cingulate cortex and premotor cortex that have not specifically been investigated here. ³¹ As the density of commercially available fNIRS systems increases, more opportunities to image a larger portion of the cortex arise. For example, a preprint by Sherafati et al ¹⁰⁷ has used HD-DOT to compare cortical activations in cochlear implant users and normal-hearing adults, allowing high-resolution imaging across a wider scalp area at varying depths of cortical penetration.

There are also some inconsistencies across currently published fNIRS studies of listening effort, most notably in conflicting results regarding the relationship between behavioral indicators of effort (e.g., reaction time, accuracy, self-reported effort) and neural indicators of listening effort. For example, higher levels of cortical activity were found for correct versus incorrect trials in normal-hearing adults by Lawrence et al ¹³⁹ and Defenderfer et al, ¹⁴² but a significant relationship between accuracy and neural activity was not found in children by Lawrence et al ¹⁴⁰ and Mushtaq et al. ¹⁴¹ White and Langdon ¹⁴⁶ found a correlation between self-reported effort and accuracy, but neither measure correlated with neural activity. Some of these inconsistencies are likely due to variability in types of listening challenge, task demands, and subject pool. ¹⁵¹ However, they may also be due to variability in analysis and statistical techniques. As fNIRS is a relatively newer technology in auditory cognitive neuroscience, there is a wide variety of approaches to analysis, and little consensus on best practices for collecting and analyzing data. Some of the studies described here analyzed concentrations of HbO, HbR, and/or HbT. Others quantified response amplitude after statistical techniques were used to regress out sources of physiological noise (e.g., beta weights after a general linear model was applied), and still others utilized transformations of the concentration data with a hemodynamic modality separation algorithm so that activation was quantified relative to silence. Each technique varies in complexity and has different implications for the conclusions drawn about cortical activation. ¹⁵² As the field of work in fNIRS progresses and best practices are established, more consistency in the techniques used to collect and analyze fNIRS data should help clarify the relationship between neural and behavioral indicators of listening effort.

Several opportunities exist for future research in listening effort using fNIRS. The variability of acoustic and linguistic challenges used in the papers reviewed here highlight the many possibilities to investigate interactions of different types of listening challenge, and how those interactions may affect activity in regions like left IFG and DLPFC. Importantly, as fNIRS is uniquely compatible with hearing instruments, new insights are possible in understanding how use of a cochlear implant or hearing aid impacts listening effort as a function of listening challenge. There is great potential for longitudinal work to be completed pre- and post-cochlear implantation, or with/without use of hearing aids, to better understand how hearing instruments affect listening effort. Additionally, an important component of listening effort that was emphasized by Pichora-Fuller et al ⁴ but not directly studied in any current work in fNIRS is the role of motivation in the listener. Paradigms that intentionally manipulate motivation at various levels of acoustic and/or linguistic challenge may shed further light on the role of nontemporal regions in listening effort, such as the cingulo-opercular network. More multimodal studies that interface fNIRS with fMRI, EEG, and/or MEG can further connect prior findings from those neuroimaging modalities with hemodynamic signatures of listening effort detected by fNIRS. Both multimodal and standalone fNIRS also have potential for future work analyzing resting-state or task-based functional connectivity between frontal, temporal, and parietal regions of interest that may be associated with behavioral data relating to listening effort. Finally, the portability of many fNIRS systems presents opportunities for more naturalistic listening situations outside of the laboratory. Bell et al ¹⁰⁹ and Rowland et al ¹⁴⁴ provided some preliminary evidence for what that may look like with simulated naturalistic acoustic environments and scenes, but fNIRS also has the capability to be used for testing in real-world challenging listening environments and situations, such as real-time conversation with a partner in a cafe, museum, or park.

Conclusion

Research using fNIRS is rapidly increasing, with the number of citations doubling every year. ⁵¹ fNIRS has many advantages for auditory cognitive neuroscience, including its compatibility with hearing instruments, resilience to motion, and silent or near-silent imaging. fNIRS has been used successfully in published research to corroborate results of previous studies of listening effort undertaken with more established neuroimaging techniques. More importantly, fNIRS has begun to be used in novel paradigms and settings that would have previously been untenable. There is much exciting opportunity for growth in the area of listening effort, and several unique features of fNIRS position it as an ideal technique for the field.