Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults

Significance

Following sensory deprivation, the sensory brain regions can become colonized by the other intact sensory modalities. In deaf individuals, evidence suggests that visual language recruits auditory brain regions and may limit hearing restoration with a cochlear implant. This suggestion underpins current rehabilitative recommendations that deaf individuals undergoing cochlear implantation should avoid using visual language. However, here we show the opposite: Recruitment of auditory brain regions by visual speech after implantation is associated with better speech understanding with a cochlear implant. This suggests adaptive benefits of visual communication because visual speech may serve to optimize, rather than hinder, restoration of hearing following implantation. These findings have implications for both neuroscientific theory and the clinical rehabilitation of cochlear implant patients worldwide.

Abstract

It has been suggested that visual language is maladaptive for hearing restoration with a cochlear implant (CI) due to cross-modal recruitment of auditory brain regions. Rehabilitative guidelines therefore discourage the use of visual language. However, neuroscientific understanding of cross-modal plasticity following cochlear implantation has been restricted due to incompatibility between established neuroimaging techniques and the surgically implanted electronic and magnetic components of the CI. As a solution to this problem, here we used functional near-infrared spectroscopy (fNIRS), a noninvasive optical neuroimaging method that is fully compatible with a CI and safe for repeated testing. The aim of this study was to examine cross-modal activation of auditory brain regions by visual speech from before to after implantation and its relation to CI success. Using fNIRS, we examined activation of superior temporal cortex to visual speech in the same profoundly deaf adults both before and 6 mo after implantation. Patients’ ability to understand auditory speech with their CI was also measured following 6 mo of CI use. Contrary to existing theory, the results demonstrate that increased cross-modal activation of auditory brain regions by visual speech from before to after implantation is associated with better speech understanding with a CI. Furthermore, activation of auditory cortex by visual and auditory speech developed in synchrony after implantation. Together these findings suggest that cross-modal plasticity by visual speech does not exert previously assumed maladaptive effects on CI success, but instead provides adaptive benefits to the restoration of hearing after implantation through an audiovisual mechanism.

A cochlear implant (CI) is an auditory prosthesis that provides a sensation of hearing to deaf individuals by electrically stimulating spiral ganglion cells of the auditory nerve. In deaf individuals, auditory regions of the brain that usually process sound can become responsive to visual stimuli (1). This cross-modal plasticity within auditory cortex can provide adaptive benefits such as superior visual localization and motion detection abilities (2). On the other hand, cross-modal plasticity can limit a deaf individual’s ability to understand speech after their hearing is restored with a cochlear implant (3, 4). Therefore, it is assumed that this maladaptive cross-modal activation of auditory brain regions must decrease following cochlear implantation for speech understanding to be restored successfully (4). However, in recent years, this traditional dichotomous stance on the adaptive effects of cross-modal plasticity during sensory deprivation versus its maladaptive effects during sensory restoration has been highlighted as too simplistic (5). For instance, it has been proposed that receiving visual linguistic input in the absence of auditory input may not necessarily limit the recovery of auditory function following implantation but instead could promote and maintain typical functioning of language networks, which could thus provide benefits for future CI outcome (5–7). However, these remain speculations as little empirical evidence exists regarding how cross-modal activation of auditory brain regions by visual speech (lip-reading) affects CI success (6, 7).

Existing evidence from a PET study in adult CI users showed that greater activation of auditory brain regions during lip-reading predicted poorer speech understanding abilities with a CI (8) and that this activity reduced from an earlier to a later stage of CI rehabilitation (9). Subsequently, it has been assumed that activation of auditory cortex by visual language can limit its capacity for auditory processing (3) and that a reduction in cross-modal activation of auditory cortex to visual speech after implantation may be crucial for successful hearing restoration (9). Such assumptions have led to clinical recommendations for deaf individuals undergoing cochlear implantation to avoid the use of visual language to maintain the ability of auditory brain regions to process auditory speech and thereby optimize CI success. However, these assumptions are currently unsubstantiated (6): how cross-modal activation of auditory brain regions by visual speech changes from preimplantation to postimplantation, and how this relates to the ability to understand speech with a CI, has yet to be investigated. Furthermore, the relationship between this postimplant cortical plasticity within auditory brain regions and the ability of these regions to respond auditory speech stimulation remains unexplored.

Preoperative brain imaging of cochlear implant users is possible using techniques such as fMRI, which has been used to understand neural mechanisms that may underlie functional CI outcomes. For instance, maintenance of typical phonological processing pathways in postlingually deaf CI candidates, as revealed by a written word rhyming task performed before implantation, has been linked to better future CI outcome (10). However, because CI devices are generally incompatible with established neuroimaging techniques including fMRI, the ability to study preimplant to postimplant cross-modal plasticity underlying hearing restoration with a CI has been severely limited (7). Here we overcame these technical challenges by using an emerging optical technique, functional near-infrared spectroscopy (fNIRS), which offers full compatibility with CI devices (11) and is safe for repeated testing. This enabled us to directly examine changes in cross-modal activation of auditory brain regions by visual speech from before to after cochlear implantation and its relation to CI success.

In line with the traditional dichotomous view of cross-modal plasticity and the available evidence, we hypothesized that a decrease in cross-modal activation of auditory brain regions by visual speech after implantation would be linked to better auditory speech understanding with a cochlear implant. Second, we investigated whether the ability of auditory brain regions to respond to sound following implantation depended on a reduction in cross-modal activation of these same regions by visual speech. We hypothesized that a decrease in cortical activation to visual speech after implantation would be linked to an increase in activation to auditory speech.

Results

Cross-modal activation of auditory brain regions during a visual speech task (lip-reading) was measured in 15 profoundly deaf individuals before cochlear implantation (T0) and 6 mo after cochlear implantation (T1). Fig. 1 displays the aggregate sensitivity profiles for our regions of interest (ROIs), illustrating the regions of bilateral superior temporal cortex (STC) to which our measurements were theoretically sensitive.

Fig. 1.

Sensitivity profiles for cortical regions of interest. Left hemisphere measurement channels (9, 10, and 12) and right hemisphere measurement channels (20, 21, and 23) are highlighted. Color scale depicts relative sensitivity to hypothetical cortical activation logarithmically from 0.001 to 1. Reprinted with permission from ref. 12.

For each individual, we first examined how cross-modal activation of auditory brain regions by visual speech changed from preimplantation to postimplantation. The direction and magnitude of change in cross-modal activation varied across the group: nine CI users displayed a decrease in activation, whereas the remaining six displayed an increase. The change in cross-modal activation was negatively correlated with the duration of bilateral hearing loss (r = −0.58, P < 0.05, two-tailed; Fig. S1), with more recently deafened individuals tending to show an increase in cross-modal activation from preimplantation to postimplantation and individuals with a longer duration of deafness tending to show a decrease. This suggests that an individual’s clinical history of deafness may influence how the brain adapts following cochlear implantation. Perhaps unsurprisingly given this level of individual variability, there was no significant change in bilateral STC activation to visual speech at the group level from preimplantation to postimplantation (Fig. 2A).

Fig. S1.

Relationship between change in cross-modal bilateral STC activation and duration of deafness. Change in cross-modal activation of bilateral STC by visual speech (∆ beta weight; arbitrary units) from T0 to T1 is plotted against the duration of bilateral hearing loss before implantation (years), with regression line shown.

Fig. 2.

Group-averaged amplitude of cross-modal activation before and after implantation. Group-averaged amplitude of cross-modal activation of STC by visual speech (in beta weight) of (A) bilateral STC, (B) left STC, and (C) right STC. Inset cortical images illustrate the sensitivity profile for the cortical regions of interest. *P < 0.05 main effect of time when assessed across both groups combined, based on the estimated marginal means from the linear mixed model analysis. n.s., nonsignificant. Error bars represent ±1 SE. CI, cochlear implant users; NH, normal-hearing controls; T0, preimplantation; T1, postimplantation.

Linear mixed model analysis of the data show that (i) there was no significant change in bilateral STC activation to visual speech over time across both CI users and normal hearing (NH) controls (no main effect of time; F_1,28.88 = 1.90, P = 0.18; Fig. 2A), (ii) there was no significant difference in cortical activation between CI users and NH controls across time points (no main effect of group; F_1,34.79 = 0.98, P = 0.33), and (iii) changes in activation to visual speech over time did not differ between the two groups (no group–time interaction; F_1,28.88 = 0.69, P = 0.41).

A significant reduction in cross-modal activation to visual speech has previously been documented from approximately 1 wk to 8 mo post-CI within anterior portions of the right superior temporal sulcus (9). Thus, we next examined changes in the amplitude of cross-modal activation to visual speech within the left and the right STC separately. Although there was no significant change in cross-modal activation of the left STC from preimplantation to postimplantation (no main effect of time; F_1,31.07 = 0.09, P = 0.76; Fig. 2B), a significant change in cross-modal activation over time was indeed observed within the right STC (main effect of time; F_1,30.01 = 6.47, P < 0.05; Fig. 2C). This indicates that the amplitude of cross-modal activation to visual speech within right STC decreased significantly over time when assessed across both groups combined.

Data pertaining to changes over time in activation of auditory brain regions by visual speech are not available from existing studies for both CI users and NH control subjects (9). We therefore asked whether the observed change over time in right STC activation to visual speech differed between CI users and NH controls. The analysis shows that pre- to post-CI changes in right STC activation did not significantly differ between the two groups (no significant main effect of group, F_1,27.18 = 1.09, P = 0.31, nor a group–time interaction, F_1,30.01 = 0.49, P = 0.49). The absence of a significant group–time interaction demonstrates that the observed change in activation of right STC to visual speech over time was not specific to the CI group and so cannot be attributed to the implantation process. However, the test–retest reliability of fNIRS responses to visual speech has been shown to be relatively poor over a retest interval of 3 mo, particularly in the right hemisphere (12). Therefore, it is possible that modest test–retest reliability prevented us from detecting a group–time interaction effect.

Auditory speech understanding 6 mo after cochlear implantation ranged from 1 to 100% correct, with a mean performance of 71% correct (SD = 33.2). The large range of CI outcomes that we observed, as well as the mean performance, is consistent with previous reports from large-scale, international studies (13–15), indicating that the CI outcomes observed in the present study may be considered representative of the wider CI population.

To identify whether a reduction in cross-modal activation of auditory brain regions by visual speech was necessary for a successful outcome following cochlear implantation, we performed a within-subject analysis to examine the relationship between change in STC activation from preimplantation to postimplantation and speech understanding with the CI. There was a strong positive correlation between change in bilateral STC activation to visual speech and speech understanding (r = 0.77, P < 0.01, two-tailed; Fig. 3). Separate correlation analysis of the left and right STC confirmed that this relationship was not driven predominantly by one cerebral hemisphere (left STC: r = 0.63, P < 0.05; right STC: r = 0.73, P < 0.01, both two-tailed; Fig. S2 A and B, respectively). Thus, contrary to expectations we found that the best performing CI users showed an increase in cross-modal activation by visual speech from preimplantation to postimplantation, whereas the poorest performing CI users showed a reduction in cross-modal activation over time. Because the change in bilateral STC activation to visual speech was associated with the duration of deafness (Fig. S1), we also examined the relationship between cross-modal plasticity and CI outcome while controlling for duration of deafness. Partial correlation analysis indicated that the observed strong positive correlation between change in bilateral STC activation to visual speech from preimplantation to postimplantation and speech understanding with a CI remained after controlling for the effect of duration of deafness (r = 0.70, P < 0.01, two-tailed).

Fig. 3.

Relationship between change in cross-modal STC activation and speech understanding. Change in cross-modal activation of bilateral STC by visual speech (Δ beta weight; arbitrary units) from T0 to T1 is plotted against speech understanding at T1 (RAU), with the regression line shown.

Fig. S2.

Change in cross-modal activation of left and right STC and speech understanding. Change in cross-modal activation of STC by visual speech (Δ beta weight; arbitrary units) from T0 to T1 is plotted against speech understanding (RAU) at T1, with the regression line shown for (A) the left hemisphere and (B) the right hemisphere.

It has been assumed that visual language may compromise the ability of auditory brain regions to respond to sound after implantation (3, 16) and that maladaptive cross-modal plasticity must be reversed for CI success (4). To explore the mechanisms underlying hearing restoration, we examined whether an increase in responsiveness of auditory brain regions to auditory speech stimulation after implantation was dependent on a decrease in cross-modal activation to visual speech. Contrary to expectations, we found a positive correlation between change in bilateral STC activation to auditory speech and change in cross-modal activation to visual speech from T0 to T1 (r = 0.51, P < 0.05, two-tailed; Fig. 4). This relationship between the auditory and visual modality did not exist in the NH control group (r = 0.09, P = 0.74, two-tailed; Fig. S3). The positive relationship seen between the two sensory modalities in the CI group contradicts the popular, yet simplistic and unsubstantiated, theory of a visual-to-auditory sensory shift within auditory brain regions from preimplantation to postimplantation. Rather, they provide evidence of an audiovisual coupling, whereby the responsiveness of auditory brain regions to auditory speech increases in synchrony with their responsiveness to visual speech from preimplantation to postimplantation.

Fig. 4.

Change in cross-modal STC activation and auditory responsiveness. Change in cross-modal activation of bilateral STC by visual speech from T0 to T1 (Δ beta weight; arbitrary units) is plotted against change in bilateral auditory responsiveness from T0 to T1 with the regression line shown.

Fig. S3.

Change in cross-modal STC activation and auditory responsiveness in control subjects. Change in cross-modal activation of bilateral STC by visual speech from T0 to T1 (Δ beta weight; arbitrary units) is plotted against change in bilateral auditory responsiveness from T0 to T1 with the regression line shown.

Discussion

Current CI rehabilitation strategies focus on hearing alone and often discourage the use of vision in the form of lip-reading (17) due to fear of an assumed adverse effect on hearing (18). Here we hypothesized that a decrease in cortical activation to visual speech after implantation would be linked to an increase in activation to auditory speech. However, the findings of this study do not support this hypothesis: longitudinal optical imaging of the human brain presented here reveals that increased cross-modal activation of auditory brain regions by lip-reading neither precludes an increase in cortical responsiveness to auditory speech, nor limits the recovery of speech understanding after implantation. Our findings in cochlear implanted adults parallel recent findings in an animal model showing that cross-modal plasticity within auditory brain regions does not preclude responsiveness to auditory stimulation with a CI and therefore should not be considered strictly maladaptive as traditionally thought (19). On the contrary, here we show that increased cross-modal activation after adult cochlear implantation is associated with increased auditory responsiveness and better speech understanding with a CI, indicating an adaptive benefit of cross-modal plasticity following implantation.

Previous postimplant imaging studies have identified subregions which differ in the direction and extent to which cross-modal STC activation to visual speech correlates with CI outcomes (8). Given the limited spatial resolution of fNIRS, it is not possible here to interrogate cortical activation in these individual subregions. Furthermore, given the large-scale averaging across millions of neurons that is inherent to all noninvasive neuroimaging techniques (and to fNIRS especially), it is not possible to classify whether it is the same population of neurons in the STC that is responding to the visual stimulus in the CI and NH groups, nor to characterize their precise nature. Therefore, although we use the term “cross-modal” to refer to putatively auditory brain regions being cross-activated by a different modality (vision), it is possible that this activation may be multimodal in its nature (i.e., reflects the activity of multisensory neurons that respond to both auditory and visual inputs). Nonetheless, despite greater spatial averaging, our findings show that changes from preimplantation to postimplantation in temporal lobe activation by visual speech are functionally relevant to CI outcome.

Our findings argue against the common view that visual language has a maladaptive effect on CI success due to cross-modal plasticity within auditory brain regions, indicating that the effects of cross-modal plasticity on sensory restoration are more complex than previously thought (5). Rather, our results provide evidence that increased cross-modal activation of auditory brain regions by visual speech may offer a facilitative link between the two modalities that promotes auditory recovery after cochlear implantation. Cross-modal activation of superior temporal cortex by visual speech may reflect processes such as inner speech and auditory imagery due to the inherent correspondence that exists between auditory and visual speech representations (20). In this way, an increase in STC activation to visual speech may reflect a stronger correspondence or synergy between the modalities that may facilitate auditory recovery. Indeed, multisensory integration of auditory and visual speech cues can enhance speech perception and is a skill shown to be enhanced in cochlear implant users compared with normal hearing individuals (21). Our finding of a synergistic link between the auditory and visual modality following cochlear implantation appears compatible with this suggestion that CI users are better multisensory integrators of auditory and visual speech cues (21). Furthermore, the regions of interest interrogated here include posterior regions of the STC, which are heavily implicated in audiovisual speech integration (22, 23). Therefore, the positive relationship observed between the two modalities here may reflect CI users’ continued reliance on visual speech cues and their integration with auditory information to decipher the degraded auditory signal provided by the implant (21, 24).

The underlying mechanisms responsible for yoking together the observed changes in responsiveness to auditory and visual stimulation within the CI group remain unclear. It has been proposed that vision may facilitate auditory perceptual learning by guiding top-down attention to auditory representations (25). As such, it is possible that changes in visual and auditory responsiveness of the STC over time may be linked through a mediating effect of top-down attention. It is also possible that the responses we measured from the STC may partly reflect generalized supramodal linguistic processing, for example, of phonological (26) or semantic information (27). Such supramodal linguistic networks may be increasingly activated by both audition and vision, as an individual CI patient learns to optimally integrate auditory and visual information to maximize language understanding. In an animal model, vision has been shown to play a facilitative role in restoring sound localization abilities after cochlear implantation (28). In parallel, our findings provide unique evidence in humans for a synergistic relationship between audition and vision within auditory brain regions, indicating a facilitative mechanism between the modalities that underlies the restoration of speech understanding following cochlear implantation.

Materials and Methods

Participants.

The study was approved by the Nottingham 1 Research Ethics Committee (reference: 12/EM/0016) and was sponsored by Nottingham University Hospitals National Health Service (NHS) Trust (Research & Innovation reference: 11IH007). All participants gave written informed consent before taking part. Common inclusion criteria across both groups were native English speakers, self-reported normal or corrected-to-normal vision, at least 18 y of age, and able to travel to and take part in all study assessments. Exclusion criteria were any known language, cognitive, or motor disorder or previous brain injury.

CI Users.

We recruited 17 adults with bilateral profound deafness who had consented to, but had not yet received, their CI device. The group included two prelingually, three perilingually, and 12 postlingually deaf individuals who were heterogeneous in their clinical characteristics (Table 1), as is typical of individuals presenting across CI clinics. All participants met United Kingdom national guidelines for cochlear implantation and had been deemed suitable CI candidates by the Nottingham Auditory Implant Program. All participants were implanted unilaterally with a Cochlear Nucleus 6 device with CP910 sound processor that used the advanced combination encoder stimulation strategy (SI Materials and Methods for further clinical information). One CI user was excluded from all analyses due to excessive motion and poor contact between fNIRS optodes and the scalp, resulting in poor data quality. Another CI user was withdrawn from the study at T1 for unrelated medical reasons.

Table 1.

Clinical characteristics of CI users

Subject ID	Age	Onset	Duration	CI side	CI T1	HA T0	HA T1	CI outcome	Handedness
CI_01	52	51	10 mo	Right	6.1	Left	Yes	97	Right
CI_02	37	Birth	37	Right	7.1	Bilateral	Yes	61	Right
CI_03	67	44	23	Right	6.2	None	No	91	Right
CI_04*	64	24	40	Left	6.1	Bilateral	Yes	92	Right
CI_05	59	20	39	Right	6.4	Left	No	97	Right
CI_06	38	Birth	38	Right	6.4	Bilateral	Yes	10	Right
CI_07	50	25	25	Right	5.3	Bilateral	Yes	99	Right
CI_08	60	52	8	Left	6.0	Bilateral	Yes	100	Left
CI_09	78	45	33	Right	5.7	Bilateral	No	93	Right
CI_10	70	30	40	Left	6.1	Left	No	64	Right
CI_11	57	3	54	Right	6.0	Right	No	85	Right
CI_12	64	5	59	Left	6.0	Bilateral	Yes	28	Left
CI_13	36	4	32	Right	6.5	None	No	1	Right
CI_14*	76	65	11	Left	—	Right	—	—	Left
CI_15	43	42	4 mo	Left	6.1	Left	No	88	Right
CI_16	78	43	35	Left	6.1	Bilateral	No	67	Right
CI_17	53	25	28	Right	6.0	Bilateral	Yes	95	Right
Mean (SD) n = 15	56.6 (13.9)				6.1 (0.4)			71 (33.2)

Age, age at implantation (years); onset, age at onset of bilateral hearing loss (years); duration, duration of bilateral hearing loss (years, unless otherwise specified); CI side, side of cochlear implantation; CI T1, duration of CI use at T1 since activation of CI device (months); HA T0, side of hearing aid worn during testing at T0; HA T1, contralateral hearing aid worn during testing at T1; CI outcome, auditory speech understanding (% correct) at T1; handedness, dominant hand.

^*CI users not included in correlational analyses.

Control Subjects.

Seventeen NH adults were recruited to serve as a control group. All participants had normal hearing thresholds, defined here as average pure-tone air conduction hearing thresholds of ≤20 decibels (dB) across frequencies 0.5, 1, 2, and 4 kHz in both ears. Audiometric testing was conducted at the beginning of each participant’s first study visit. The recruitment of control subjects was staggered in an attempt to approximately match the group’s mean age (57 y ±16.8) to that of the CI users (58.2 y ±13.9). Due to attrition, one NH control subject did not complete testing at T1.

Experimental Design.

A longitudinal repeated-measures design was used. The same neuroimaging and behavioral tests were administered to all participants at two time points. For CI users, the first testing session (T0) took place at their earliest convenience after having consented to receive a CI but before undergoing surgery (preimplantation). At T0, CI users were tested in their best-aided condition, i.e., wearing their hearing aids if they used them in everyday conditions. The second testing session (T1) was conducted approximately 6 mo after activation of the CI (postimplantation, average duration of CI use = 6.1 mo, SD = 0.4). At T1, CI users were tested in their best-aided condition wearing their preferred listening devices (i.e., CI and optional contralateral hearing aid). The mean retest interval between T0 and T1 was 8.2 mo (SD = 1.2).

NH control subjects similarly underwent testing in two sessions. The T0–T1 retest interval was set to mirror that of the CI group as closely as was pragmatically possible, given the variation in clinical waiting times for the CI operation and device activation. The mean retest interval between T0 and T1 was 8.1 mo (SD = 0.3).

Testing Conditions.

Testing was carried out in a double-walled sound-attenuated booth. Participants were seated in front of a visual display unit (VDU) at a viewing distance of 1 m. Visual components of the stimuli were presented on the VDU. To reflect the typical level of conversational speech, auditory components were presented through a centrally located loudspeaker at 65 dB sound pressure level (SPL; A-weighted root-mean-square level averaged over the duration of each sentence). See SI Materials and Methods for further information.

fNIRS Scanning.

In each testing session, cortical activation was measured using a continuous-wave fNIRS system (ETG-4000; Hitachi Medical Co.). The ETG-4000 is a commercial system that emits a continuous beam of light into the cortex and samples at a rate of 10 Hz. The system measures simultaneously at two wavelengths, 695 and 830 nm, to allow for the separate measurement of changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) concentrations. This specific choice of wavelengths has been shown to minimize cross-talk error between the two chromophores (29).

fNIRS Stimuli.

The Institute of Hearing Research (IHR) Number Sentences (20) were presented as speech stimuli during the acquisition of fNIRS measurements. The corpus comprised digital audiovisual recordings of 90 sentences, each spoken by both a male and female talker. Each of the sentences contained between four and seven words, three of which were designated keywords. For the purpose of this experiment, the speech material was presented in two stimulation conditions: (i) auditory-only (A-ONLY) where the auditory component was presented but the visual component was not shown and (ii) visual-only (i.e., lip-reading, V-ONLY) where the visual component of the recording was shown but the auditory component was muted. The speech material was also presented in an audiovisual condition (auditory and visual components presented congruently) for the purpose of a separate experiment to be reported elsewhere. In the A-ONLY condition the background remained uniform and a fixation cross was presented in place of the talker’s mouth. Rest periods consisted of this uniform background and fixation cross only.

fNIRS Paradigm.

Thirty IHR number sentences were randomly selected without replacement for presentation in each of the conditions, with the restriction that an equal number were spoken by the male and female talker in each condition. The speech stimuli were presented in a block-design paradigm interleaved with rest periods. Each block comprised six concatenated sentences, evenly spaced to fill a 24-s block duration. Five blocks were presented for each stimulation condition. During these blocks, the participants were instructed to attend to the talker and to always try to understand what the talker was saying. To encourage sustained attention to the experimental stimuli, an attentional trial was presented after 2 of the 15 stimulation blocks. These blocks were chosen at random, and therefore, the attentional trials occurred at unpredictable positions within the experimental run. Two seconds after the cessation of a chosen block, two alternative words were presented on either side of the fixation cross; in a two-alternative forced-choice task, participants were asked to press one of two buttons to indicate which word had been spoken in the immediately preceding sentence. Following the participant’s response, an additional 5-s rest was added to the start of the ensuing rest period. Rest periods were included to allow the hemodynamic response elicited by the stimulation block to return to a baseline level. The durations of the rest periods were randomly varied between 20 and 40 s in 5-s increments. Before fNIRS scanning, participants first completed a short familiarization run to ensure that they understood the experimental procedure (SI Materials and Methods for further details).

Optode Placement.

Two 3 × 3 optode arrays were placed bilaterally over the subject’s temporal lobes. The optode arrays were positioned on the participant’s head so as to ensure good coverage of the STC (Fig. 1 and Fig. S4). Optode positioning was guided by the International 10–20 System (30) to promote consistency across participants and test sessions (SI Materials and Methods for further details).

Fig. S4.

Mean position of fNIRS optodes and measurement channels. Measurement channels are labeled numerically, source optodes are indicated in red, and detector optodes are indicated in blue.

Definition of ROI.

To assess the sensitivity of our fNIRS measurements to the underlying cortical regions, using the AtlasViewer tool (31) a Monte Carlo code for simulating the probabilistic path of photon migration through the head (32) (tMCimg) was run with 1 × 10⁷ simulated photons launched from each optode position. The resultant sensitivity profiles (Fig. 1) suggested that channels 9, 10, and 12 (left hemisphere) and channels 20, 21, and 23 (right hemisphere) provided appropriate sensitivity to the posterior portion of STC. Therefore, these measurement channels were predefined as the left and right superior temporal ROIs. The left and right ROIs together formed the bilateral STC ROI.

Behavioral Test of Speech Understanding.

The City University of New York (CUNY) Sentence Lists (33) were used to obtain a measure of speech understanding (SI Materials and Methods for further details). The CUNY Sentence Lists include 25 standardized lists each comprising 12 sentences that vary in length and topic. Each list contains between 101 and 103 words spoken by a male talker.

For the purpose of this experiment, two CUNY lists (i.e., 24 sentences) were randomly selected without replacement for presentation in the A-ONLY stimulation condition. Speech understanding in V-ONLY and audio–visual modalities was also tested for the purpose of a separate experiment to be reported elsewhere. The 24 sentences were presented in random order. After each sentence presentation, the participant was instructed to repeat back all words that they were able to identify. All words correctly reported by the participant were recorded by the researcher on a scoring laptop before initiation of the next trial. The scoring method ignored errors of case or declensions. Before commencement of speech understanding testing, all participants completed a short familiarization run (SI Materials and Methods).

Processing of fNIRS Data.

Raw fNIRS recordings were exported from the Hitachi ETG-4000 into MATLAB for use with routines provided in the HOMER2 package (34) and custom scripts. To prepare the recordings for subsequent analyses they were subjected to a set of preprocessing steps, including motion artifact correction, bandpass filtering, and hemodynamic signal separation. Full details of all preprocessing steps are provided in SI Materials and Methods. To quantify the level of cortical activation, the preprocessed fNIRS signal was subjected to an ordinary least squares (OLS) general linear model (GLM). The GLM design matrix included three boxcar regressors, one for each of the stimulation conditions. The two response periods following the two attentional trials were also modeled in the design matrix as isolated events occurring at the time the two words were presented on screen. These were convolved with the canonical hemodynamic response function provided in Statistical Parametric Mapping software version 8 (www.fil.ion.ucl.ac.uk/spm). After completing the first-stage OLS estimation at the single-subject level, we used the Cochrane–Orcutt procedure (35) to correct for serial correlation. Briefly, this involved fitting a first-order autoregressive process to the model residuals and transforming the original model according to the estimated autoregressive parameter (see ref. 36). We then reestimated the beta weights based on the transformed model (second stage).

The beta weights of the canonical hemodynamic response function term were extracted at each measurement channel, for each stimulation condition, and for all participants. The hemodynamic signal separation method used here (37) (SI Materials and Methods) assumes a fixed linear relationship between HbO and HbR in the functional response. Therefore, the results of all statistical analyses are identical regardless of whether conducted on the beta weights extracted for the HbO or HbR parameter. For simplicity, only results pertaining to the beta estimates of the HbO parameter of the functional component are presented here. These beta weights were used to quantify the amplitude of cortical activation for each condition compared with rest. The resultant beta weights were averaged across the ROI measurement channels for each group and at each time point and were subjected to further statistical analysis as outlined below.

Processing of Behavioral Data.

Speech understanding, measured using the CUNY Sentence Lists, was quantified as the percentage of words reported correctly (% correct). To make the data more suitable for statistical analysis, the rationalized arcsine transform (38) was applied using MATLAB (SI Materials and Methods for details). Subsequently, the transformed scores [rationalized arcsine units (RAUs)] were subjected to statistical analysis.

Statistical Analysis.

Following the preprocessing of neuroimaging and behavioral data, resultant data were analyzed and figures were produced using IBM (International Business Machines Corporation) Statistical Package for the Social Sciences (SPSS) Statistics software (Release 22.0; IBM Corp.). Data and analysis scripts are publically available through the University of Nottingham’s Research Data Management Repository (https://rdmc.nottingham.ac.uk).

Linear Mixed Model Analysis.

The ROI beta weights were analyzed separately for the bilateral, left, and right ROI using a linear mixed model (LMM; see SI Materials and Methods for further information). Each model included two fixed factors of group and time to estimate the fixed effect of experimental group (CI users versus NH controls) and time relative to implantation (T0, before implantation; T1, 6 mo after CI activation) on cross-modal activation. In addition, a group–time interaction term was specified to understand whether an effect of time on cortical activation differed between the two groups. Specifically, if a group–time interaction indicated that cross-modal activation changed over time in the CI group but remained comparatively stable in the NH group, this would suggest an effect specific to the CI process.

Correlational Analysis.

Change in amplitude of cross-modal activation from preimplantation to postimplantation was calculated as the difference between the amplitude (beta weight) of STC activation to visual speech measured at T0 and T1. Bivariate correlation analysis was conducted to examine the nature of the relationship between change in cross-modal activation (Δ beta weight) and speech understanding (RAU). Specifically, the parametric statistic Pearson’s correlation coefficient (r) was used to estimate the direction and strength of the linear relationship. Similarly, Pearson’s correlation was conducted to examine the direction and strength of the relationship between change in cross-modal activation and change in amplitude of STC activation to auditory speech (auditory responsiveness).

SI Materials and Methods

CI Users.

None of the participants experienced any complications during their CI surgery, and no abnormalities were identified on postoperative X-ray. For all participants, all implantable electrodes were situated within the cochlea, and postoperative impedances were within normal range on all electrodes. For one participant, activation of the device was delayed by 1 wk due to an infection around the surgical site of incision. All participants were stimulated in monopolar configuration, and comfort and threshold levels were estimated for each electrode position by the clinical team according to standard clinical protocols.

Testing Conditions.

All stimuli were presented, and scored where appropriate, using the MATLAB computing environment (Release 2014b; The MathWorks). A centrally located Genelec 8030A loudspeaker was mounted immediately above and behind the visual display unit. The SPL of auditory stimuli was measured at the listening position with the participant absent using a Brüel & Kjær 2250 sound level meter and free-field microphone (Type 4189). A dense sound-absorbing screen was placed between the fNIRS equipment and the participant to attenuate the steady fan noise generated by the equipment. This resulted in a steady ambient noise level of 38 dB SPL (A-weighted). The fNIRS equipment remained switched on, although not recording, throughout behavioral testing. Before the commencement of each test, participants were provided with written instructions to ensure understanding and consistency of instructions given.

Familiarization Runs.

During the fNIRS familiarization session, one block of each of the conditions was presented. To avoid preexposure to the experimental stimuli, the familiarization blocks comprised speech material [BKB (Bamford–Kowal–Bench) sentences (39)] that were different from the material presented during the fNIRS measurements and the subsequent behavioral testing. Following each stimulation block, an example of the attentional control task was also presented. Similarly, BKB sentences were used during the behavioral familiarization run to avoid preexposure to the CUNY corpus.

Optode Placement.

Together, the optode arrays comprised 10 emitter and 8 detector optodes with a fixed interoptode distance of 30 mm, providing a penetration depth into the cortex of ≈15 mm (40). This resulted in a total of 24 measurement channels (12 per hemisphere). Optode positioning was guided by the International 10–20 System (30) to promote consistency across participants and test sessions. Specifically, on each side, the lowermost source optode was placed as close as possible to the preauricular point, with the uppermost source optode aligned toward Cz. Consistency of optode positioning across test sessions at the individual level was further ensured by reference to photographs taken during the initial testing session.

To evaluate the consistency of optode positioning across individuals, the procedure was piloted on six adult volunteers who did not take part in the main experiment. After positioning the arrays as described above, the optode positions, plus anatomical surface landmarks, were recorded using the Hitachi ETG-4000s electromagnetic 3D Probe Positioning Unit. For each volunteer, the digitized optode positions were registered to a standard atlas brain (41) (Colin27) using the AtlasViewer tool (31), allowing their locations to be visualized relative to underlying cortical anatomy. The SD in the position of each optode was between 2.9 and 8.8 mm. Assessment of the mean optode positions suggested that the array provided good coverage of STC (Fig. S4).

Behavioral Test of Speech Understanding.

The CUNY corpus was used primarily due to its routine use as a clinical outcome measure by CI programs across the United Kingdom. Additionally, this corpus was not presented during fNIRS scanning, thus helping to limit training effects within and across testing sessions.

Processing of fNIRS Data.

Raw light intensity measurements were first converted to change in optical density (34). Wavelet motion correction was then performed to reduce the effect of motion artifacts on the fNIRS signal. Wavelet filtering can enhance data yield and has emerged as a favorable approach for use with fNIRS data (42). The HOMER2 hmrMotionCorrectWavelet function (based on ref. 42) was used, which assumes that the wavelet coefficients have a Gaussian probability distribution and so applies a probability threshold to remove outlying wavelet coefficients that are assumed to correspond to motion artifacts. A probability threshold was set to exclude coefficients lying more than 1.5 interquartile ranges below the first quartile or above the third quartile.

Following motion artifact correction, a bandpass filter of 0.01–0.5 Hz was applied to reduce sources of physiological noise in the data including high-frequency cardiac oscillations and low-frequency respiration and blood pressure changes. The fNIRS signal was next converted into estimates of changes in HbO and HbR using the modified Beer–Lambert law with a default differential path length factor of six (34). Because bandpass filtering is unable to remove all physiological noise from fNIRS recordings (34), the hemodynamic signal separation method of (37) was also applied. This algorithm separates the fNIRS signal into estimates of the functional and systemic components, based on expected differences in the correlation between HbO and HbR in each component. Specifically, a positive correlation between changes in HbO and HbR is assumed in the systemic component, whereas a negative correlation is assumed in the functional component. The functional component of the signal was identified by the algorithm, extracted from the fNIRS signal and retained for further analysis.

Processing of Behavioral Data.

The arcsine transform (T) was applied as follows:

$$T=\text{arcsine}\sqrt{\frac{X}{N+1}}+\text{arcsine}\sqrt{\frac{X+1}{N+1}}.$$

The asin function in Matlab was used to return the inverse sine (arcsine) for each value of X, where X represents the total number of words reported correctly and N represents the total number of words presented. This was then transformed linearly:

$$R=46.47324337T-23,$$

where R indicates the resulting rationalized arcsine-transformed score (RAU). This transformation extends the original percent correct scale outwards in both directions from 50%, creating bigger differences as the extremes of the range are approached. Consequently, this transformation makes the rationalized arcsine scale linear and additive in its proportions while producing values close to that of the original percentage scores (38).

Linear Mixed Model Analysis.

LMMs are able to give accurate estimates of the fixed effect of predictor variables while incorporating and accounting for the random effects that are often inherent in repeated-measures designs (43). Missing data were missing for reasons that were unrelated to the observed data values (i.e., attrition and unrelated medical condition). The LMMs were therefore conducted under the assumption that missing data were missing at random (43). Thus, the LMMs maximized the use of available data rather than entirely omitting subjects with missing data values from the analysis.

Due to the repeated measures design, participant was specified as a variable to incorporate within-participant covariance into the fixed effect estimation. Furthermore, a random intercept effect and random effect of time were also specified to facilitate the estimation of the fixed effect of group and time while accounting for possible between-subject variability in overall fNIRS response amplitude and the trajectory of change over time.