Auditory and audio–visual processing in patients with cochlear, auditory brainstem, and auditory midbrain implants: An EEG study

Abstract

There is substantial variability in speech recognition ability across patients with cochlear implants (CIs), auditory brainstem implants (ABIs), and auditory midbrain implants (AMIs). To better understand how this variability is related to central processing differences, the current electroencephalography (EEG) study compared hearing abilities and auditory–cortex activation in patients with electrical stimulation at different sites of the auditory pathway. Three different groups of patients with auditory implants (Hannover Medical School; ABI: n = 6, CI: n = 6; AMI: n = 2) performed a speeded response task and a speech recognition test with auditory, visual, and audio–visual stimuli. Behavioral performance and cortical processing of auditory and audio–visual stimuli were compared between groups. ABI and AMI patients showed prolonged response times on auditory and audio–visual stimuli compared with NH listeners and CI patients. This was confirmed by prolonged N1 latencies and reduced N1 amplitudes in ABI and AMI patients. However, patients with central auditory implants showed a remarkable gain in performance when visual and auditory input was combined, in both speech and non‐speech conditions, which was reflected by a strong visual modulation of auditory–cortex activation in these individuals. In sum, the results suggest that the behavioral improvement for audio–visual conditions in central auditory implant patients is based on enhanced audio–visual interactions in the auditory cortex. Their findings may provide important implications for the optimization of electrical stimulation and rehabilitation strategies in patients with central auditory prostheses. Hum Brain Mapp 38:2206–2225, 2017. © 2017 Wiley Periodicals, Inc.

INTRODUCTION

Patients with sensorineural hearing loss can regain hearing with cochlear implants (CIs), which transform the acoustic signal into electrical pulses and stimulate the auditory nerve [Wilson and Dorman, 2008]. Many CI recipients achieve open‐set speech understanding, and many early implanted children develop age‐appropriate language skills [Geers et al., 2009; Geers and Nicholas, 2013; Kral and O'Donoghue, 2010]. However, a CI is not effective in patients whose auditory nerve is destroyed or congenitally absent. In those patients, auditory brainstem implants (ABIs) or auditory midbrain implants (AMIs) can restore some auditory function. In these implants, stimulation is provided either by a flat array of electrodes positioned on the surface of the cochlear nucleus [ABI; see, e.g., Lenarz et al., 2001; Wilkinson et al., 2014], or by a penetrating multi‐channel electrode placed in the central nucleus of the inferior colliculus [AMI; see, e.g., Lenarz et al., 2006; Lim et al., 2009].

So far over 1,200 patients worldwide have received an ABI [Shannon, 2015], and in a first clinical trial, five patients have been implanted with an AMI at the Hannover Medical School [Lenarz et al., 2007; Lim et al., 2009]. Patients with central auditory implants typically show low speech perception. Only a few ABI patients approach performance levels of CI patients [Behr et al., 2014; Colletti et al., 2012; Lenarz et al., 2001; Matthies et al., 2014]. The variability and limitation in outcomes with central auditory implants may have different causes: Neural damage might emerge due to the surgery, the electrode array may be placed inaccurately [Colletti et al., 2012], or the growth of bilateral cochleovestibular schwannomas in patients with neurofibromatosis type 2 (NF2) may cause damage in the cochlear nucleus [Behr et al., 2014]. Moreover, central auditory implants do not yet have suitable electrode array designs and the used CI processing strategies are not optimized for stimulation of more centrally located nervous structures [Lim and Lenarz, 2015; McKay et al., 2013].

Efforts to understand and improve speech perception in patients with central auditory implants are of utmost importance. To better understand the special requirements for electrical stimulation, previous radiotracer imaging studies exploring the cortical response patterns in CI, ABI, and AMI patients could show bilateral recruitment of the auditory cortex to speech and non‐speech sounds in all groups of patients [Berding et al., 2015; Coez et al., 2009; Miyamoto et al., 1999; Miyamoto and Wong, 2001; Di Nardo et al., 2001, 2004]. First studies have proven the feasibility of recordings of electrically evoked auditory brainstem, mid‐latency and cortical auditory responses in ABI [He et al., 2015; Herrmann et al., 2015; O'Driscoll et al., 2011] and AMI patients [Colletti et al., 2007]. A next important step is to directly compare the electrophysiological correlates of cochlear, auditory brainstem, and auditory midbrain implant patients. Here, results could reveal central processing differences dependent on the site of stimulation.

As a clinical tool, auditory event‐related potentials (AEPs) can be used to objectively measure auditory function and rehabilitation in patients with auditory implants. In congenitally deaf children with CIs, measures of the P1 and N1 component can be used to monitor maturation of the auditory system, which has been shown to be restricted by a sensitive period, closing around 3.5 years of age [for review, see Sharma et al., 2015]. Similarly, AEPs in CI children have been served to identify a maximal time window of less than 1.5 years delay for sequential bilateral CI implantation to guarantee normal development of the auditory system [for review, see Gordon et al., 2013]. Studies in post‐lingually deafened adult CI patients showed AEPs to be an important tool to monitor rehabilitation processes, during which N1 latency has been shown to reduce and N1 amplitude to increase [Sandmann et al., 2015].

Previous studies with central auditory implant patients have mainly focused on the auditory modality. However, real‐world situations, such as the conversation with other people, typically involve the stimulation of different senses concurrently. Similar to CI patients, individuals with central auditory implants may develop multisensory strategies to overcome the limited input from the implant [Schierholz et al., 2015]. It is reasonable to assume that multisensory interactions are even more important in these patients compared with CI patients, given that more centripetally located brain regions are damaged or inadequately stimulated. The benefit obtained from combined stimulus presentation and its dependence on the site of stimulation remains unexplored.

Therefore, the present study used electroencephalography (EEG) to examine event‐related potentials (ERPs) to auditory, visual, and audio–visual stimuli in CI, ABI, and AMI patients, as well as in a control group of normal‐hearing (NH) listeners. The ultimate novelty of this study is the comparison of auditory and audio–visual processing across patients, covering the whole spectrum of today's available auditory neural prostheses. Previous results showed prolonged latencies and reduced amplitudes in CI patients compared with NH listeners [Agrawal et al., 2013; Groenen et al., 2001; Kelly et al., 2005; Sandmann et al., 2009, 2015; Viola et al., 2011]. Moreover, the N1 has been observed to be associated with speech recognition ability, both with regard to latency [e.g., Finke et al., 2016] and amplitude [Sandmann et al., 2015]. In ABI and AMI patients, the target structures are not yet stimulated optimally. Accordingly, the even more suboptimal input from central auditory implants is hypothesized to result in longer latencies and reduced amplitudes of AEPs in ABI/AMI patients when compared with CI patients and NH listeners.

METHODS

Participants

Fourteen volunteers took part in the present study. Half of the participants (n = 7) were post‐lingually deafened auditory brainstem implant (ABI) patients from the Hannover Medical School, with six of them being implanted unilaterally (3 right‐, 3 left‐implanted). One ABI patient had an additional AMI contralateral to the ABI. As there was a substantial age variance across the ABI patients, seven sex‐ and aged‐matched NH listeners served as control group. One ABI patient (with the contralateral AMI) had to be excluded from the later analysis, because of a noise‐contaminated EEG in combination with a highly prominent electrical artifact from the implant. To keep the groups of participants equal, the exclusion of the ABI patient lead to the exclusion of the match from the NH group. Thus, for the final analysis 12 individuals were included: 6 ABI patients (3 female, 3 right‐implanted, mean age: 57.0, range: 34–72, standard error of the mean (SEM): 6.1 years, Table 1) and 6 NH listeners (3 female, mean age: 59.0, range: 35–74, SEM: 6.1 years). Additionally, we re‐analyzed the recently published data of 6 post‐lingually deafened, unilaterally implanted CI patients [Schierholz et al., 2015]. The group of CI patients was sex‐ and age‐matched (3 female, 3 right‐implanted, mean age: 55.2, range: 21–76, SEM: 8.8 years, Table 1) to the groups of ABI patients and NH listeners.

Table 1.

Subject demographics of implant patients

	Sex	Age	Handedness	Implant side	Contralateral to implant side	Etiology	Age at onset of profound deafness (years)	Duration of deafness (months)	Implant use (months)	Implant	Open‐set word recognition (%)			GainSpeech	Subjective benefit
											A	V	AV
CI1	M	21	Right	Right	Deaf	Meningitis	7	2	167	Nucleus M‐24	75	–	–	–	9
CI2	F	37	Right	Right	Hearing Aid	Acute HL	24	19	140	AB_Clarion CII	75	25	95	20	9
CI3	M	62	Right	Left	Hearing Aid	Acute HL	55	70	15	Medel_Concerto Standard	90	0	85	−5	8
CI4	M	64	Right	Right	Residual Hearing	Cholesteatoma	48	153	35	Nucleus_CI512	95	5	95	0	5
CI5	F	71	Ambidexter	Left	Hearing Aid	Unknown	58	108	50	AB_HiRes90K	90	0	80	−10	10
CI6	F	76	Right	Left	Hearing Aid	Unknown	71	26	25	Nucleus CI24RE Contour Advance	65	10	80	15	6
ABI1	F	34	Right	Right	Deaf	Self‐mutilation	31	16	26	Nucleus ABI M‐24	30	10	50	20	–
ABI2	M	44	Right	Right	Deaf	NF2	32	0	147	Nucleus ABI M‐24	5	10	45	40	4
ABI3	M	59	Ambidexter	Left	Deaf	NF2	37	90	176	Nucleus ABI M‐24	0	15	35	35	2
ABI4	F	63	Left	Left	Deaf	Meningitis	51	41	102	Nucleus ABI M‐24	0	30	25	25	5
ABI5	M	70	Right	Right	Deaf	NF2	50	19	225	Nucleus ABI 22	0	5	70	70	5
ABI6	F	72	Right	Left	Deaf	Meningitis	53	93	136	Nucleus ABI M‐24	10	25	45	35	4
AMI1	F	33	Right	Left	Deaf	NF2	22	51	84	Nucleus AMI 24RE	0	5	10	10	4
AMI2	F	50	Right	Right	Deaf	NF2	38	47	100	Nucleus AMI 24RE	10	5	35	25	4

Note. F, Female; M, Male; NF2, Neurofibromatosis type 2; HL, Hearing loss; A, Auditory; V, Visual; AV, Audio–visual. “Duration of deafness” refers to the time period between the “age at onset of profound deafness” and the time point of implantation. The Gain_Speech in word recognition was calculated by means of the following equation: [Performance AV − Performance A]. The subjective benefit of the implant was assessed by participant ratings of how speech understanding is experienced in daily communication situations. The rating was achieved by means of an evaluation on an 11‐point scale ranging from “very difficult” (0) to “very easy” (11).

Finally, we examined two female auditory–midbrain implant (AMI) patients from the Hannover Medical School (1 right‐implanted, age range: 33–50 years, Table 1). Due to the small sample size, the results of these two patients will be reported only descriptively. All patients used their implant constantly for at least 12 months. Duration of deafness was defined by the time period between the “age at onset of profound deafness” and the date of device implantation. Information about the demographic variables and the implant systems of the patients are provided in Table 1. Patient groups did not significantly differ regarding the duration of deafness, the implant experience and the age at implantation (all P ≥ 0.18). Cognitive status of the participants was proven to be in the norm by the clock completion test [Watson et al., 1993] and the Regensburg word fluency test [lexical and semantic; Aschenbrenner et al., 2000]. None of the NH participants had a record of psychiatric illness. All NH participants had age‐appropriate hearing (age < 50 years: 0.25–8 kHz ≤ 15 dB; age > 60 years: 0.25–2 kHz ≤ 35 dB, 4–8 kHz ≤ 50 dB hearing loss in the tested ear), and normal or corrected‐to‐normal vision, verified by means of a Snellen chart with letters (mean: 81%, SEM: 5%). Similar to a previous study [Stropahl et al., 2015], we assessed speech recognition scores in patients and controls by using monosyllabic words. They were taken from the Freiburg monosyllabic word test [Hahlbrock, 1970], which is often used in the clinical routine. Stimuli were recorded as audio–visual videos (AVI video format; 25 frames/s; 720 × 576 pixel) with a professional male speaker and presented in three conditions (V: video track only, A: audio track only, AV: audio and video track; in quiet, 65 dB SPL). The spoken words began one second after the video onset and, for videos with visual component, each word started and ended in a neutral position with the mouth closed. While this test has not been validated yet, to our knowledge no validated test is currently available for the parallel investigation of auditory, visual, and audio–visual speech recognition.

All participants gave written informed consent prior to data collection and were reimbursed. The present study was carried out in accordance with the Declaration of Helsinki principles and was approved by the Ethics Committee of the Hannover Medical School.

Stimuli and Procedure

The stimuli and the procedure were identical to those used in our previous study with CI patients [see Schierholz et al., 2015]. The paradigm included stimuli of three different conditions: modality‐specific visual (V), modality‐specific auditory (A), and cross‐modal audio–visual (AV). The stimuli were delivered using the Presentation software (Neurobehavioral Systems, version 16.5) and a personal computer connected with a 27 inch monitor (1,920 × 1,080 × 32 bit, 60 Hz screen refresh rate). The stimulus of the modality‐specific visual condition consisted of a white disk (diameter: 0.5° of visual angle) on a grey background, appearing for 50 ms either to the right or to the left side of a central fixation cross, at a peri‐foveal eccentricity of 4.0° visual angle. The luminance of the white disk (303.1 cd/m², Konica Minolta: LS 100) and the background (36.7 cd/m²) resulted in a Michelson contrast of 78.4%. The stimulus of the modality‐specific auditory condition consisted of a 1 kHz sinusoidal tone (Adobe Audition CS6, version 5.0.2), sampled at 44.1 kHz and 50 ms in duration. For implant patients, auditory stimulation was provided via two loudspeakers (HECO victa 301) positioned at 50° azimuth. If the patients had an additional device (hearing aid, implant) contralateral to the tested ear, it was detached for the time of the experiment. In all patients, the contralateral ear was occluded by means of a wax ear‐plug. NH listeners received monaural auditory stimulation via insert earphones (3M E‐A‐RTONE 3A) at the ear consistent with the stimulation side of the matched implant patient. Auditory stimulation was provided with approximately 65 dB SPL. Each participant evaluated the loudness of the auditory stimulus by use of a seven‐point loudness‐rating scale [Sandmann et al., 2009, 2010, 2015] in order to adjust the perceived loudness to a moderate level [∼60–70 dB(A); Allen, 1990; Zeng, 1994]. The cross‐modal audio–visual stimulus was made up of the concurrent presentation of the visual (50% left, 50% right screen side) and auditory stimulus [at the implanted (CI, ABI, AMI)/matched (NH) ear]. Additionally, a “nostim” condition was included in the experiment (see section “EEG Preprocessing” for further details) which consisted of a fixation cross presented in the center of the screen.

Participants were seated comfortably in a dimly lit and electromagnetically shielded booth at a viewing distance of 155 cm to the screen. Individuals were instructed in writing to indicate the detection of any stimulus—irrespective of its modality—as fast as possible and by means of a button press with the dominant hand. They were further instructed to not respond in case of no event (nostim) and to maintain fixation on the center of the screen for the entire task. Each condition (V, A, AV, nostim) contained 120 trials [total number: 480 (120 repetitions × 4 conditions)]. Trials started with a fixation cross at the center of the screen, which—except for trials of the nostim condition—was accompanied by a modality‐specific visual (V), modality‐specific auditory (A), or cross‐modal audio–visual (AV) stimulus, after a jittered interval of 405–905 ms. The fixation cross lasted for the entire trial until the end of the response window (950 ms duration). Before the start of the experiment, each participant accomplished a mixed practice block including 24 trials (8 trials per condition). An outline of the paradigm is shown in Figure 1A.

Figure 1.

(A) Simplified design of the paradigm. Note, periods between two consecutive stimuli are not displayed (for details see method section). (B) Predefined cluster of electrodes used for individual identification of the auditory electrode ROI. The individual auditory ROI consists of seven electrodes including the electrode with the individual maximal N1 peak and six neighboring electrodes.

EEG Recording

EEG data were continuously recorded by use of 94 Ag/AgCl electrodes incorporated in a customized, infracerebral electrode cap with an equidistant electrode layout (Easycap, Herrsching, Germany). For the recording, three linked 32‐channel BrainAmp amplifiers (BrainProducts, Gilching, Germany) were used. An electrooculogram was recorded by two additional electrodes positioned below the eyes. A nose‐tip electrode was used as reference, and a midline electrode, positioned slightly anterior to Fz, served as ground. Data recording was performed using a sampling rate of 1,000 Hz and an online analog filter from 0.02 to 250 Hz. Impedances were maintained below 10 kΩ prior to data acquisition.

Data Analysis

For the data analysis MATLAB 8.1.0.604 (R2013a; Mathworks, Natick, MA) and SPSS 23.0 (IBM, Armonk, NY) were used. In general, we used non‐parametric tests for the statistical analysis due to the small sample size. If not stated otherwise, P‐values report exact significance. We considered P‐values of 0.05 or below as statistically significant. Bonferroni correction was applied where appropriate.

Speech recognition scores

Speech recognition scores represent the percentage of words, repeated correctly in the respective condition (V, A, AV). To test for differences between groups (NH, CI, ABI), Kruskal–Wallis tests were performed separately for each condition.

We further analyzed the difference between the performance in the auditory‐only and the audio–visual condition [word recognition score (AV) – word recognition score (A)]. Thereby, we evaluated the benefit in word recognition by the presence of additional visual information (i.e., lip movement). The computed difference score is referred to as redundancy gain (Gain_Speech) and was compared between groups by means of a Kruskal–Wallis test.

Speeded response task

False alarm responses (mean: 2.0, SEM: .8%) and trials with very fast reaction times (RTs < 100 ms; mean: .9, SEM: .3%) were excluded for the analysis. Moreover, outlier trials (RTs exceeding the individual mean by more than 3 standard deviations) were excluded for each individual participant and condition (V: mean: 1.0, SEM: .2%; A: mean: 1.5, SEM: .2%; AV: mean: 0.8, SEM: 0.2%). Due to restrictions in trial number for the CI patients of the previous study, only 60 trials of the cross‐modal condition—in detail those with visual stimulation on the right side—were included in the analysis [for more information, see Schierholz et al., 2015]. Therefore, the analysis of visual‐only and audio–visual trials was restricted to the 60 trials with right visual stimulation in all groups (NH, CI, ABI, AMI) to avoid confounds and to ensure comparability between conditions and groups. For the auditory condition all available trials were included.

Hit rates and RTs for each condition (V, A, AV) were compared between (Kruskal–Wallis tests) and within groups (Wilcoxon signed‐rank tests). As for the speech recognition scores, we also analyzed the redundancy gain in RTs (Gain_RT), that is, the difference in RTs between the audio–visual and the auditory condition [RT (AV) – RT (A)]. Thereby, we evaluated the benefit in task performance by the presence of additional visual information, compared with the auditory‐only condition. The Gain_RT was compared between groups (Kruskal–Wallis test).

In a next step, separately for each group, we tested for the manifestation of the redundant signals effect, that is, a facilitation in RTs to redundant cross‐modal information compared with modality‐specific information [Miller, 1982]. This was done by comparing the audio–visual condition and the fastest modality‐specific condition (V or A, respectively) within each group by means of one‐sided Wilcoxon signed‐rank tests.

Furthermore, we tested for the violation of the race model inequality [RMI; Miller, 1982] to resolve whether the faster RTs observed on redundant signals can be explained by statistical facilitation [race model; Raab, 1962] or by a coactivation model. Briefly, the race model acts on the assumption of a “race” of competing independent unisensory processing ways [Raab, 1962]. Accordingly, RTs on redundant signals are predicted to be faster due to statistical facilitation, because the probability of either of two stimuli to produce a fast RT is larger than that from one single stimulus alone. In contrast, the coactivation model [Miller, 1982] proposes an interaction of the neural responses to the single stimuli of a pair. Accordingly, neural responses are combined and form a new product, thus leading to faster RTs. We applied the RMI to resolve whether the observed faster RTs on redundant signals are based on statistical facilitation or on coactivation:

$$P\left({\text{RT}}_{\text{AV}}\le t\right)\le P\left({\text{RT}}_{\mathrm{A}}\le t\right)+P\left({\text{RT}}_{\mathrm{V}}\le t\right), \text{for} \text{all} t\ge 0,$$

where P(RT_x ≤ t) denotes the probability of a RT in condition x to be below an arbitrary value t. Violation of the model, for any given value of t, points to the presence of multisensory interactions [see also Ulrich et al., 2007 for detailed information on the application of the RMI]. Using the RMITest software [Ulrich et al., 2007], cumulative distribution functions (CDFs) of the RT distributions were estimated for each recorded condition (V, A, AV), as well as for the sum of the modality‐specific conditions (A + V), individually for each participant. RTs of the individual participant were rank ordered for each condition to determine percentile values [Ratcliff, 1979]. Given that the difference between the redundant signals condition (AV) and the modality‐specific sum (A + V) for the five fastest deciles (width: 10%) was proven to be normally distributed (Shapiro–Wilk tests), subsequently, one‐tailed t‐tests were used to determine if the difference between the two conditions (AV, A + V) was significantly different from zero. Significance at any decile bin was treated as violation of the race model.

EEG preprocessing

EEG data analysis was performed with MATLAB (Mathworks, Natick, MA) and EEGLAB [version 12.0.2.5b; Delorme and Makeig, 2004]. Raw data were down‐sampled (500 Hz), re‐referenced (common average reference), bandpass filtered [1–40 Hz, FIR filter, Hann window; Widmann and Schröger, 2012] and epoched into 2 s segments. Subsequently, data were pruned of unique, non‐stereotype artifacts and an infomax independent component analysis (ICA) was computed [Bell and Sejnowski, 1995]. Successively, the resulting ICA weights were applied to the original raw data, bandpass filtered from 0.5 to 40 Hz and epoched from −500 to 1,100 ms relative to the stimulus onset (V, A, AV, nostim). Independent components representing artifacts such as eye‐blinks, horizontal eye movements, electrical heartbeat activity, as well as other sources of non‐cerebral activity were identified and removed from the data [Jung et al., 2000a, 2000b]. To optimize the identification and correction of the electrical artifacts of the implant, a second infomax ICA was performed. Here, epochs were shortened (−100 to 400 ms, relative to the auditory onset) to concentrate on the time window where the electrical artifact is expected to occur. In a next step, independent components, reflecting artifacts of the implant, were identified by the centroid on the side of the device, as well as the time course of the component activity. After removal of these components, missing channels in the proximity of the transmitter coil and the speech processor were interpolated (mean: 6 electrodes; SEM: .4; range: 4–9).

As for the analysis of the behavioral data, 60 trials for the audio–visual and the visual‐only condition—specifically those with the visual stimulus on the right side—were included for the analysis. Similarly, the amount of trials in the auditory‐only condition was reduced to n = 60 by random selection (implant patients and NH listeners received auditory stimulation constantly either on the right or on the left side). The reduction in the number of trials had no significant impact on the signal quality of ERPs. This was shown by a group‐specific comparison of the signal‐to‐noise ratios (SNRs) for 60 and 120 trials, computed for the auditory N1 peak at electrode Cz (NH, CI, ABI: all P ≥ 0.093).

EEG data analysis

ERPs for modality‐specific and cross‐modal conditions were compared within and between groups (NH, CI, ABI). Audio–visual interactions were evaluated using the additive model [AV = A + V; Barth et al., 1995]. In case the model is satisfied, the sum of the responses to the modality‐specific stimuli (A, V) would be equal to the response elicited by the cross‐modal (AV) stimulus. This result would argue for an independent processing of both stimuli of the cross‐modal pair. However, any violation of the assumption (AV = A + V) emphases the presence of some interaction processes. Application of the equation, though, may be problematic because of common activity (e.g., slow wave potentials) which will appear once on the left side, but twice on the right part of the equation. This can lead to an artifactual interaction effect [Teder‐Sälejärvi et al., 2002]. To avoid artificial discrepancy in the equation, the time‐locked average of the nostim condition is added to the left part of the equation [AV + nostim = A + V; see, e.g., Hauthal et al., 2015; Mishra et al., 2007, 2010; Schierholz et al., 2015; Senkowski et al., 2011; Talsma and Woldorff, 2005]. The rearrangement of the equation (A = AV + nostim – V) provides the possibility to specifically compare the recorded response to modality‐specific auditory stimuli (A_uni) and the term [AV + nostim – V], referred to as the visually modulated auditory response (A_mod). Here, A_mod represents the estimated auditory response in an audio–visual context, that is, the auditory response modified by the concurrent processing of a visual stimulus. Any difference between A_uni and A_mod is taken as proof for the presence of multisensory interactions.

We defined the time window (68–200 ms) for analysis of the N1 peak of unisensory (A_uni) and modulated (A_mod) AEPs by visual inspection of the grand average AEP. Similar to a recent study, electrodes used for the peak detection analysis were selected individually within a pre‐defined region‐of‐interest (ROI, Fig. 1), covering fronto‐central scalp regions [Chen et al., 2015]. This was done to account for the interindividual variability in the specific channel with maximal N1 deflection, as it is known that ERP topographies can differ due to individual differences in brain size, detailed anatomy and function, even when the activation originates from the same brain source [Luck, 2014]. Even in invasive recordings, exact maps within the primary auditory cortex vary substantially in position, and details of sulcal patterns are variable between individual animals [Merzenich et al., 1975]. The same is true for cytoarchitectonic maps [Rose, 1949]. Also in humans several studies provided evidence for a substantial interindividual variability in areal location [Morosan et al., 2005; Rademacher et al., 2001; Zilles et al., 1997]. The use of the traditional approach, using the same electrodes for each participant, therefore, might be a confound by ignoring individual differences.

Within our predefined ROI, we identified an individual ROI for each participant, enclosing the electrode with the maximal N1 peak and six neighboring electrodes. Subsequently, auditory responses—separately for the unisensory and the modulated response—were quantified by computing the mean peak amplitude (±4 ms) around the local minimum (N1) of the AEP in the pre‐defined time window and ROI (peakdet.m; http://www.billauer.co.il/peakdet.html). Peak latency was defined as the time of the local minimum. Similar to previous observations, we expected greater N1 amplitudes for the modulated compared with the unisensory response [Schierholz et al., 2015]. This was tested by means of a one‐tailed Wilcoxon signed‐rank test, separately in each group. Furthermore, latencies for unisensory and modulated responses were compared within each group (Wilcoxon signed‐rank test). In addition, amplitudes and latencies of unisensory and modulated responses were compared between groups (Kruskal–Wallis tests).

Additionally, we compared the visual event‐related potentials (VEPs) to the unisensory visual stimulation between the different groups. Here, Kruskal–Wallis tests were used to compare mean amplitudes (±4 ms around the local maximum/minimum) and peak latencies of the P1 and the N2 VEPs between NH listeners, CI and ABI patients. The time windows were set to 64–144 ms (P1) and 130–250 ms (N2). Similar to the AEPs, electrodes used for the peak detection analysis were selected individually within a pre‐defined region‐of‐interest, enclosing the occipital scalp region. Within this predefined area, an individual ROI was determined for each participant, enclosing the electrode with the maximal N2 peak and six neighboring electrodes. The N2, but not the P1 was chosen for determination of the individual ROI, as VEPs at N2 latency showed a higher signal‐to‐noise ratio.

Source analysis

The standardized low resolution brain electromagnetic tomography software (sLORETA; publicly available free academic software at http://www.uzh.ch/keyinst/loreta.htm) was used to examine the activation in the auditory cortex during the unisensory and the visually modulated auditory processing. Based on the scalp‐recorded AEPs, sLORETA computes the cortical three‐dimensional distribution of the current source density. Compared with other source analysis techniques, sLORETA is advantageous, as it does not need any assumptions about the number and position of sources [Pascual‐Marqui, 2002]. Because of the smoothness constraint, the sLORETA source solutions have a low spatial resolution. Nevertheless, the method has the characteristic of exact localization [Sekihara et al., 2005]. Similar to previous studies, we restricted the analysis to cerebral electrodes (n = 83) to optimize the source estimations [Gottselig et al., 2004; Schierholz et al., 2015].

Source estimates were computed for the AEP grand average (NH, CI, ABI, and AMI) at N1 latency. To optimize these results, two procedures were applied prior to the source estimation: First, the AEP averages of the single subjects were aligned with respect to the N1 peak to obtain maximal coincidence in time. Second, the data of the participants with right‐sided auditory stimulation (NH: n = 3, CI: n = 3, ABI: n = 3) were flipped such that electrodes on the right side changed to the respective position on the left side and vice versa. This was done based on the prediction that unilateral stimulation in NH listeners, CI and ABI patients would result in input mostly to the cortex contralateral to the stimulation side [Gilley et al., 2008; Di Nardo et al., 2001, 2004; Sandmann et al., 2015]. With respect to the AMI patients, the data of the patient with left‐sided (but not right‐sided) stimulation was mirrored, as the anatomical connections of the inferior colliculus mainly project to the ipsilateral cortex [e.g., Casseday et al., 1976; Kudo and Niimi, 1980; Malmierca, 2004; Moore and Goldberg, 1963; Oliver, 1984]. The severe hearing impairment may have altered the hemispheric asymmetry in our participants, given that previous studies have reported reduced hemispheric asymmetry in individuals and animals with unilateral and bilateral deafness [Fujiki et al., 1998; Kral et al., 2009, 2013; Ponton et al., 2001]. However, contralateral dominance for auditory processing has been previously reported in CI and ABI patients [Gilley et al., 2008; Di Nardo et al., 2001, 2004; Sandmann et al., 2015]. Further, to our knowledge there is no study comparing the hemispheric asymmetry across different groups of auditory‐implant patients. Thus, it is currently unknown whether the contralateral dominance differs between CI, ABI and AMI patients.

After having completed these two steps, the grand average AEP was imported into sLORETA. Source estimation around the N1 peak (peak ± 20 ms) revealed a current density maximum in the temporal lobe, especially in the Brodmann Area (BA) 41, indicating maximal activation in the auditory cortex (Fig. 5).

Figure 5.

Auditory cortex activation for unisensory auditory (A_uni, black) and visually modulated auditory responses [A_mod (AV + nostim‐V), red]. (A) NH listeners, (B) CI patients, (C) ABI patients, (D) mean N1 amplitude values (± SEM) for NH listeners, CI and ABI patients, (E, F) single AMI patients. Note, the gray area marks the time window used for peak detection. The result of the source estimation (N1 peak ± 20 ms), shown in the lower left part of the figure, revealed maximal activation in the temporal lobe, particularly in Brodmann Area (BA) 41. The auditory ROI (BAs 22, 41, and 42 of both hemispheres), used for the source analysis, is shown in the lower right part of the figure. Asterisks indicate significance (*P < 0.05, ** P < 0.01). [Color figure can be viewed at http://wileyonlinelibrary.com]

In a following step, we carried out a region‐of‐interest (ROI) analysis. For this analysis, the original single‐subject data, without flipping or alignment, were taken. An auditory ROI was defined, enclosing the BAs 22, 41, and 42, covering the auditory cortex of both, the left and the right hemisphere. The BAs were selected based on the result of our source estimation, as well as on previous reports that the primary auditory cortex is the source of neural activity at N1 latency [Hari et al., 1980; Hine and Debener, 2007; Näätänen and Picton, 1987; Sandmann et al., 2009]. Moreover, multisensory interactions have been shown to involve BA 22 [Beauchamp, 2005; Sperdin et al., 2009].

As the stimulation side differed between participants, the source waveforms, obtained from the ROI analysis, were averaged over the left and right auditory cortex. Consistent with the procedure on the sensor level, quantification of the activity in the auditory ROI was carried out for the time window of 68–200 ms after stimulus onset. Values of the mean amplitude (±4 ms around the local maximum) and the latency of the N1 peak for the unisensory (A_uni) and the modulated (A_mod) responses were subjected to statistical analysis. As for the sensor level, the hypothesis of greater N1 amplitudes in the auditory cortex for the modulated compared with the unisensory responses was tested by means of one‐tailed Wilcoxon signed‐rank tests. Latencies for unisensory and modulated responses were compared within groups (Wilcoxon signed‐rank tests) and both, amplitudes and latencies for the unisensory and the modulated responses were compared between groups (Kruskal–Wallis tests).

RESULTS

Speech Recognition: CI and ABI Patients

Speech recognition scores obtained in the three different conditions are shown in Table 2 and Figure 2A. Group differences were found for the auditory (H(2) = 15.38, P < 0.001) and the audio–visual (H(2) = 14.27, P < 0.001), but not for the visual condition (P = 0.42). Post‐hoc Mann–Whitney U tests comparing NH listeners and CI patients revealed significantly higher speech recognition scores for the NH listeners in both the auditory (U = 0.50, P < 0.05, r = −0.84) and the audio–visual condition (U = 1.00, P < 0.05, r = −0.82). Similarly, NH listeners showed significantly higher speech recognition scores than ABI patients in both the auditory (U = 0.00, P < 0.01, r = −0.87) and the audio–visual condition (U = 0.00, P < 0.01, r = −0.86). Comparing CI and ABI patients, results revealed significantly better speech recognition scores for the CI patients in both the auditory (U = 0.00, P < 0.01, r = −0.84) and the audio–visual condition (U = 0.00, P < 0.05, r = −0.83).

Table 2.

Mean speech recognition scores for monosyllabic words (%) ± one standard error of the mean (SEM)

	NH	CI	ABI	AMI 1	AMI 2
Visual	13 ± 6	8 ± 5a	16 ± 4	5	5
Auditory	99 ± 1	82 ± 5	8 ± 5	0	10
Audio–visual	99 ± 1	87 ± 3a	45 ± 6	10	35
GainSpeech (AV – A)	0.0 ± 1.3	4.0 ± 5.8a	37.5 ± 7.2	10	25

^a n = 5. Note, in these conditions only data of five CI patients were available.

Figure 2.

(A) Speech recognition scores for monosyllabic words for the visual (V, blue), auditory (A, red) and audio–visual (AV, green) condition, displayed separately for each group. In general, note, in the AMI group only two datasets were obtained. Digits refer to the single AMI patients (1 = AMI1, 2 = AMI2). Inferential statistics was only computed for the first three groups (NH, CI, ABI). Error bars indicate the standard error of the mean (SEM). (B) Audio–visual gain in speech recognition (Gain_Speech) for all four groups. The gain was computed as the difference in performance between the audio–visual and the auditory condition (AV‐A). (C) Mean response times of the redundant target paradigm for the visual (V, blue), auditory (A, red) and audio–visual (AV, green) condition, for all groups. (D) Audio–visual gain in response times (Gain_RT), separately for all groups. The gain was computed as the difference in response times between the audio–visual and the auditory condition (AV‐A). In general, asterisks indicate significance (*P < 0.05, ** P < 0.01). [Color figure can be viewed at http://wileyonlinelibrary.com]

One‐sided one sample Wilcoxon signed rank tests showed that ABI patients (P < 0.05) but not NH listeners or CI patients (both, P > 0.05) showed an audio–visual gain in speech recognition (Gain_Speech; Table 2, Fig. 2B) significantly greater than zero. This difference was underlined by a significant group difference (H(2) = 10.99, P < 0.01). Post‐hoc Mann–Whitney U tests showed a significantly stronger Gain_Speech in ABI patients compared with NH listeners (U = 0.00, P < 0.01, r = −0.85), as well as compared with CI patients (U = 0.50, P < 0.05, r = −0.80). The Gain_Speech of NH listeners and CI patients was not significantly different.

Speech Recognition: AMI Patients

Speech recognition scores of the AMI patients are documented in Tables 1 and 2, and Figure 2A. Patient AMI1 did not reveal any word recognition for the auditory only condition. Scores for the visual and the audio–visual condition were slightly higher. Similarly, patient AMI2 showed a low performance for the modality‐specific conditions. However, for the audio–visual condition, the performance was clearly superior compared with patient AMI1. The relative gain in performance from unisensory auditory to audio–visual stimulation (Gain_Speech) was greater in patient AMI2 than in patient AMI1 (Table 2, Fig. 2B).

Performance in Speeded Response Task: CI and ABI Patients

Overall, the performance was high (hit rate ≥95%) in all task conditions (Table 3). Comparisons of hit rates revealed no significant difference within or between groups in any of the conditions.

Table 3.

Mean of hit rates (%) and response times (ms) ± one standard error of the mean (SEM)

		NH	CI	ABI	AMI 1	AMI 2
Hit rates	Visual	98.0 ± 0.7	96.0 ± 2.1	95.0 ± 3.0	70	98
	Auditory	97.0 ± 1.0	98.0 ± 0.7	95.0 ± 2.9	82	95
	Audio–visual	98.0 ± 0.9	99.0 ± 0.4	95.0 ± 3.1	92	98
Response times	Visual	332 ± 14	333 ± 19	382 ± 24	522	439
	Auditory	289 ± 17	308 ± 28	421 ± 11	572	365
	Audio–visual	242 ± 16	256 ± 14	324 ± 15	509	303
	GainRT (AV – A)	−47 ± 10	−52 ± 15	−97 ± 15	−63	−63

Reaction times (RTs) are shown in Table 3 and Figure 2C. Between‐subject tests showed group differences for RTs in the auditory (H(2) = 9.17, P < 0.01) and the audio–visual condition (H(2) = 9.28, P < 0.01), but not for the visual condition (P = 0.26). Comparisons of NH listeners and CI patients revealed no differences in RTs for the auditory or the audio–visual condition. ABI patients compared with NH listeners showed significantly longer RTs in both, the auditory (U = 0.00, P < 0.01, r = −0.83) and the audio–visual condition (U = 2.00, P < 0.05, r = −0.74). Similarly RTs of ABI patients were significantly longer in the audio–visual condition (U = 2.00, P < 0.05, r = −0.74) and marginally longer in the auditory condition (U = 4.00, P = 0.078, r = −0.65) when compared with CI patients.

Within‐subject comparisons revealed no significant differences between the two modality‐specific conditions in none of the groups (V vs. A; all P > 0.05). However, each group showed significantly faster RTs in the cross‐modal condition (AV) compared with the RTs of the fastest of the two modality‐specific conditions (all groups: Z = −2.20, P < 0.05, r = −0.90). Thus, each group revealed a redundant signals effect.

A significant redundancy gain in RTs (Gain_RT; Table 3, Fig. 2D) was present in all groups, as indicated by one‐sided one sample Wilcoxon signed‐rank tests against 0 (all P < 0.05). However, a difference between groups was revealed (H(2) = 6.42, P < 0.05). Post‐hoc Mann‐Whitney U tests showed a significantly stronger gain in ABI patients compared with NH listeners (U = 3.00, P < 0.05, r = −0.69). By contrast, the comparison of redundancy gains between NH listeners and CI patients, as well as between CI and ABI patients did not reach significant thresholds.

To test for the violation of the race model, the race model inequality was applied which revealed significant one sample t‐tests in at least one decile per group (Table 4). Consequently, measured RTs in the audiovisual condition were faster than those predicted by the model, as can also be observed in the cumulative distribution functions (CDFs, Fig. 3), emphasizing multisensory interactions in all groups.

Table 4.

Redundant signals and modality‐specific sum for each decile

	NH				CI				ABI
Decile	AV	A + V	ΔAV	P	AV	A + V	ΔAV	P	AV	A + V	ΔAV	P
0.10	192	201	9	0.012	198	207	9	0.055	251	271	20	0.005*
0.20	206	220	14	0.006*	215	226	11	0.020	271	289	18	0.005*
0.30	217	232	15	0.018	228	242	14	0.018	284	309	25	0.002*
0.40	226	245	19	0.011	240	255	15	0.008*	295	322	28	0.003*
0.50	236	255	19	0.024	254	267	12	0.003*	307	331	24	0.002*

Note. AV corresponds to the redundant signals condition and A + V to the modality‐specific sum. ΔAV corresponds to the difference between AV and A + V. One‐sided t‐tests with Bonferroni correction for multiple comparisons were conducted separately for each group, to test whether ΔAV was significantly different from zero. An asterisk indicates significance (*P < 0.05/5).

Figure 3.

Cumulative distribution functions, separately for groups of NH listeners, CI and ABI patients (A–C) and for the two single AMI patients (D–E), of the response times for the visual (V, blue), the auditory (A, red), and the redundant audio–visual (AV, green) condition, as well as of the modality‐specific sum (A + V, black), predicted by the race model. [Color figure can be viewed at http://wileyonlinelibrary.com]

Performance in Speeded Response Task: AMI Patients

Hit rates and RTs of the AMI patients can be observed in Table 3. Hit rates of AMI2 were high, as for the other groups. However, AMI1 showed slightly reduced hit rates and noticeably prolonged RTs compared with the other three groups and compared with AMI2 (Fig. 2C). Though, AMI1 revealed a similar RT pattern as the ABI patients, that is, faster RTs to visual compared with auditory stimuli, and fastest RTs in the audio–visual condition. By contrast, AMI2 showed a response pattern that was more similar to the group of CI patients, that is, faster RTs in the auditory compared with the visual condition, and the fastest RTs in the audio–visual condition. Similar to the other groups, both AMI patients revealed a redundant signals effect, which is consistent with the finding that both AMI patients showed a prominent Gain_RT. The Gain_RT was highly similar for the two AMI patients and lay in between those of the CI and ABI patients (Table 3, Fig. 2D).

Regarding the race model inequality, AMI2 descriptively revealed a violation of the race model, which is in line with the observations made for NH listeners, CI and ABI patients (Fig. 3). However, for AMI1 a different pattern can be observed, arguing against a violation of the race model inequality in this patient.

Electrophysiological Results: CI and ABI Patients

Grand average scalp AEPs and source waveforms (SWFs) for the unisensory (A_uni) and the visually modulated auditory response (A_mod) can be obtained in Figures 4 and 5, respectively. In each group, a prominent deflection around 100 ms, referred to as the N1 peak, can be detected in the AEPs, as well as in the source waveforms. Moreover, descriptively, AEPs/auditory source waveforms of all groups (NH, CI, ABI) showed a visual modulation, indicated by an enhanced amplitude for A_mod compared with A_uni. On the source level, the relative increase in amplitude from A_uni to A_mod was larger for the ABI (A_uni: 0.58 µA/mm²; A_mod: 1.26 µA/mm²) than for the CI patients (A_uni: 1.51 µA/mm²; A_mod: 2.96 µA/mm²), possibly demonstrating a larger benefit of audio–visual stimulation in ABI patients at the level of the auditory cortex.

Figure 4.

Grand averages of auditory event‐related potentials (AEPs), that is, for auditory unisensory (A_uni, black) and visually modulated auditory responses (A_mod (AV + nostim‐V), red). (A) NH listeners, (B) CI patients, (C) ABI patients, (D) mean N1 amplitude values (± SEM) for NH listeners, CI and ABI patients, (E, F) single AMI patients. Note, the grey area marks the time window used for peak detection. Asterisks indicate significance (*P < 0.05, ** P < 0.01). [Color figure can be viewed at http://wileyonlinelibrary.com]

Comparisons of the N1 amplitude (Table 5) between groups revealed differences for the unisensory responses, on both, the sensor and the source level (sensor: H(2) = 9.31, P < 0.01; source: H(2) = 7.61, P < 0.05); and for the modulated responses on the sensor level (H(2) = 8.57, P < 0.01). NH listeners showed larger amplitudes for the unisensory responses compared with ABI patients, on both, the sensor and the source level (sensor: U = 0.00, P < 0.01, r = −0.83; source: U = 0.00, P < 0.01, r = −0.83). Furthermore, on the sensor level, NH listeners and CI patients showed larger amplitudes for the modulated responses compared with the ABI patients (NH: U = 3.00, P < 0.05, r = −0.69; CI: U = 2.00, P < 0.05, r = −0.74). No differences were observed between NH listeners and CI patients.

Table 5.

Mean N1 latencies (ms) and N1 amplitudes (µV and µA/mm², respectively) of AEPs and source waveforms ± one standard error of the mean (SEM)

		NH	CI	ABI	AMI 1	AMI 2
AEPs	Latency unisensory	89 ± 4	112 ± 4	137 ± 10	122	164
	Latency modulated	87 ± 4	117 ± 7	137 ± 7	120	150
	Amplitude unisensory	−6.0 ± 0.4	−4.9 ± 0.6	−3.6 ± 0.4	−4.3	−2.6
	Amplitude modulated	−6.5 ± 0.5	−7.3 ± 0.9	−4.3 ± 0.5	−3.7	−4.8
Source waveforms	Latency unisensory	85 ± 2	107 ± 5	130 ± 13	124	166
	Latency modulated	81 ± 3	121 ± 7	144 ± 8	118	150
	Amplitude unisensory	2.2 ± 0.4	1.5 ± 0.5	0.6 ± 0.1	0.6	0.1
	Amplitude modulated	2.2 ± 0.4	3.0 ± 0.9	1.3 ± 0.3	0.4	0.7

Regarding the N1 latency (Table 5), between‐group comparisons revealed significant differences, for both, unisensory and modulated AEPs and SWFs (sensor: unisensory: H(2) = 12.79, P < 0.001; modulated: H(2) = 12.72, P < 0.001; source: unisensory: H(2) = 11.50, P < 0.001; modulated: H(2) = 12.67, P < 0.001). Patients with ABIs showed longer latencies for unisensory and modulated responses when compared with NH listeners (sensor: unisensory: U = 0.00, P < 0.01, r = −0.83; modulated: U = 0.00, P < 0.01, r = −0.84; source: unisensory: U = 1.00, P < 0.05, r = −0.79; modulated: U = 0.00, P < 0.01, r = −0.83). Similarly, N1 latencies for unisensory and modulated responses were prolonged in CI patients compared with NH listeners (sensor: unisensory: U = 2.00, P < 0.05, r = −0.74; modulated: U = 0.50, P < 0.05, r = −0.81; source: unisensory: U = 1.00, P < 0.05, r = −0.79; modulated: U = 0.00, P < 0.01, r = −0.83). Furthermore, ABI patients, in tendency, showed longer latencies for unisensory responses when compared with CI patients on the sensor level (U = 4.00, P = 0.072, r = −0.65).

Tests of within‐group differences showed increased N1 amplitudes on the sensor and source level for the modulated compared with the unisensory responses, in both, CI and ABI patients (sensor and source: Z = −2.20, P < 0.05, r = −0.90). By contrast, NH listeners showed no significant difference between N1 amplitudes of unisensory and modulated AEPs or SWFs. Regarding the N1 latencies, no differences were found between the unisensory and the modulated responses.

We computed correlation analyses to test for a relationship between AEP measures, behavioral performance and demographic variables. However, no significant correlations have been observed. The lack of significant relationships could be related to the small sample size and does not necessarily mean that the AEP measures are not related to behavioral measures or demographic variables.

The statistical analysis of the VEPs to unisensory visual stimulation (P1, N2) revealed no significant differences between the three groups (P1: latency: P = 0.202; amplitude: P = 0.183; N2: latency: P = 0.482; amplitude: P = 0.181).

Electrophysiological Results: AMI Patients

Electrophysiological results (Table 5, Figs. 4 and 5) revealed AEPs and auditory cortex activation for both AMI patients. In general, AEP morphology was similar to what was observed in the other groups. However, the results of the two AMI patients were dissimilar. AEPs and SWFs of AMI1 revealed highly similar N1 amplitudes to ABI patients, but N1 latencies lay somewhere in between those observed in CI and ABI patients. For AMI2, on both the sensor and the source level, N1 latencies were prolonged and the N1 amplitudes of the unisensory responses were decreased when compared with ABI patients. However, AMI2 showed a strong positive visual modulation, that is, an increase in N1 amplitude from the unisensory to the modulated response, on both, the sensor and the source level, consistent with the observations in CI and ABI patients. However, in AMI1 no positive modulation of the N1 amplitude, but rather a slight decrease from the unisensory to the modulated response was observed, on both, the sensor and the source level.

VEPs for the AMI patients were similar to those obtained for the other three groups, suggesting no difference in the processing of basic visual stimuli.

DISCUSSION

In the current study we examined the processing of auditory, visual and audio–visual stimuli in different groups of post‐lingually deafened auditory implant patients (CI, ABI, AMI), covering the whole spectrum of today's available auditory neural prostheses. This is the first study to directly compare behavioral and electrophysiological measures for electrical stimulation at different stages of the auditory pathway. Behavioral results of ABI and AMI patients revealed reduced performance in the modality‐specific auditory condition. However, most patients with central auditory implants showed a remarkable gain in performance when visual and auditory input was combined. Audio–visual interactions were confirmed in all groups of implant patients by a strong visual modulation of auditory–cortex activation. Our behavioral and EEG results demonstrate that the stimulation of more centripetal structures is associated with more difficulties to process the electrical stimuli.

Speech Recognition Ability

Open‐set speech recognition for words varied substantially between the different groups. ABI patients' speech perception scores were inferior compared with NH participants as well as compared with CI patients. Also, the speech perception varied substantially within ABI patients. Other studies have suggested that ABI patients with NF2—an autosomal dominant disorder which causes the growth of bilateral cochleovestibular schwannomas [e.g., Baser et al., 2003; Slattery, 2015]—have limited open‐set speech recognition scores that are far behind the typical performance levels of CI [Lenarz et al., 2001, 2002; Nevison et al., 2002; Otto et al., 2002; Schwartz et al., 2003] or non‐NF2 ABI patients [Colletti, 2006]. This led to the assumption that the tumor growth and/or the surgical removal of the tumor may be the primary cause of the limitations in speech recognition in patients with NF2 [Colletti, 2006; Colletti and Shannon, 2005]. However, more recent studies in NF2 ABI patients have reported advanced speech recognition skills in some patients, with levels close to those of CI patients [Behr et al., 2014; Colletti et al., 2012; Matthies et al., 2014]. The improved outcome might be the result of recent advancements in surgical procedures during tumor removal, as well as in implant placement. In our study, the majority of ABI patients was implanted more than a decade ago (median: 141.5 months), which may at least partially account for the poor speech recognition ability in this group of patients. Further, our results for patients with (n = 3) and without NF2 (n = 3) cannot confirm previous reports of inferior performance in NF2 patients. Nevertheless, based on the small sample size in the current study, we refrain from drawing firm conclusions about this observation.

Consistent with ABI patients, speech recognition ability of AMI patients was particularly low in the auditory condition. This underlines the importance of further AMI development including electrode design, stimulation strategy, and surgical procedure. In the second clinical trial, efforts are now under way to improve hearing performance in patients implanted with an AMI by reducing the effective stimulation rate per channel and by increasing the number of available stimulation channels with a double‐shank electrode [Lim and Lenarz, 2015]. With further extensive development, the AMI may turn out to be a device with a great future potential, especially for patients that cannot benefit from a CI or ABI.

In the visual condition, word recognition was comparable between NH listeners, CI and ABI patients. This indicates no significant differences in lip‐reading skills between those groups. Consistently, lip‐reading ability of AMI patients was not enhanced when compared with the other groups. Our test results on lip‐reading appear in discrepancy to previous findings with better lip‐reading abilities in CI patients than in NH subjects [Rouger et al., 2007, 2012; Stropahl et al., 2015]. Similar stimulus material, specifically monosyllabic words, was used for the visual, auditory and audio–visual conditions in the present study. Given the comparably low performance in the visual only condition across groups, the stronger audio–visual benefit for ABI patients points to a real combined effect of the visual and auditory modality, rather than a benefit in the visual condition per se in ABI patients. Furthermore, it needs to be emphasized that this outcome likely does not reflect the lip‐reading abilities in natural conditions, where more visual cues and context are available than in the present experimental condition, and where implanted subjects likely outperform NH listeners. Moreover, our CI participants were good performers (speech recognition for monosyllabic words ≥65%), and they reported to not actively use lip reading in daily life. This is consistent with our observation that CI patients scored considerably higher in the auditory compared with the visual condition, thus showing the same response pattern as NH listeners. Our findings highlight the good recovery of auditory function in CI patients and the strong reliance on auditory cues particularly in proficient CI patients [Tremblay et al., 2010]. Similar to the CI patients, ABI and AMI patients did demonstrate low levels of lip‐reading skills, although their recovery of auditory functions was not as good as for the CI patients. However, as already mentioned, for the audio–visual condition, ABI and AMI patients revealed a remarkable improvement in performance compared with both, the visual‐only and the auditory‐only condition. The observed Gain_Speech was significantly stronger in ABI patients compared with both, NH listeners and CI patients. Interestingly, for ABI patients, the detection probability in the bimodal speech condition exceeded the sum of the probabilities in the corresponding unimodal conditions [Colonius, 2015; Stevenson et al., 2014]. This is commonly interpreted as evidence for a coactivation mechanism and points to multisensory interactions in ABI patients in this speech recognition task. By contrast, NH listeners and CI patients performed already at ceiling in the auditory only condition, making it difficult to further improve speech recognition. ABI patients, on the other hand, could substantially improve their performance from the two unisensory to the multisensory speech condition, but, nevertheless, did not reach the performance levels of NH listeners and CI patients.

Performance in the Speeded Response Task

In the unisensory auditory condition of the speeded response task, CI patients showed RTs in the range of the NH listeners. These findings are in line with previous observations in CI patients, proving detection of auditory signals at the level of NH listeners [Landry et al., 2013; Schierholz et al., 2015]. However, ABI patients revealed significantly longer RTs for the detection of auditory signals compared with NH listeners. In tendency, RTs were also prolonged compared with CI patients. A general slowdown of sensory and/or cognitive processes in ABI patients can be ruled out, as RTs for visual events were comparable between NH listeners, CI and ABI patients. Prolonged RTs, relative to those in NH listeners and CI patients, only occurred in conditions conveying auditory information. This suggests a modality‐specific, auditory slowdown in ABI patients.

Consistent with the results of ABI patients, AMI patients showed prolonged RTs when compared with NH listeners and CI patients. Whereas AMI2 performed in the range of ABI patients, AMI1 showed even prolonged RTs compared with the ABI group. In sum, our findings indicate that in patients with central auditory implants the detection of simple auditory stimuli is not restored to the levels of NH listeners. This is in contrast to the fast auditory reactivity observed in CI patients. Differences between implantees could at least partly originate from the fact that central auditory implants use strategies of CIs which are not optimized for stimulation of more centrally located nervous structures. Up to now, the special requirements for electrical stimulation at the level of the brainstem or the midbrain have been considered only to insufficient extent [Lim and Lenarz, 2015; McKay et al., 2013].

Hit rates and RTs for the visual condition of the speeded response task were comparable between groups. These results are in line with former findings of similar hit rates and RTs in CI patients and NH listeners [Schierholz et al., 2015]. Noteworthy, previous studies with congenitally deaf individuals have suggested enhanced visual reactivity in these individuals compared with NH listeners [e.g., Bottari et al., 2010; Loke and Song, 1991]. By contrast, our results from post‐lingually deafened ABI and CI patients did not show exceptional visual skills. Similarly, there was no visual improvement for the AMI patients when compared with the other groups. Patient AMI2 even showed a prolongation of RTs specifically in the visual condition when compared with the other groups. Moreover, in patient AMI1, RTs were generally prolonged, independent of the condition, suggesting a general slowdown of sensory and/or cognitive processes in this participant. Taken together, our findings indicate no experience‐related, visual improvement in patients with CI, ABI or AMI, at least for simple visual stimuli in the context of a speeded response task.

The redundant signals effect (i.e., significantly faster RTs to the audio–visual compared with the fastest modality‐specific condition) was recognized in each group. This effect has been previously reported in NH listeners [e.g., Girard et al., 2011; Mahoney et al., 2011] and in CI patients [e.g., Nava et al., 2014; Schierholz et al., 2015]. Hence, our results extend previous findings by showing that a redundant signals effect also exists in patients with central auditory prostheses. The existence of a redundant signals effect does not inevitably point to the existence of multisensory integration, as this effect could also be the consequence of statistical facilitation [Miller, 1982]. However, our results showed a violation of the race model inequality in NH listeners and auditory implant patients (CI, ABI), emphasizing the enhanced performance in the AV condition to be the result of true multisensory interactions. The same conclusion applies for AMI2, who showed a violation of the race model on the descriptive level. In contrast, in AMI1 the race model was not violated. However, this does not exclude the existence of multisensory interactions, as they have been observed even in the absence of a race‐model violation [Murray et al., 2001; Sperdin et al., 2009].

Similar to the speech condition, ABI and AMI patients showed a strong audio–visual gain in RTs (Gain_RT) for basic, non‐speech stimuli. This (relative) enhancement in reactivity for audio–visual compared with auditory‐only stimuli was more pronounced in ABI and AMI patients than in CI patients and NH listeners, although patients with central auditory implants revealed slower (absolute) RTs to audio–visual stimuli than CI patients and NH listeners. Similar to the pronounced gain in speech recognition (Gain_Speech), the strong gain in reactivity (Gain_RT) might be the consequence of enhanced audio–visual interactions as form of a compensatory mechanism in ABI and AMI patients. Regarding CI patients, the Gain_RT was not as strong. This is in line with findings of earlier studies, reporting no difference in multisensory Gain_RT between post‐lingually deafened CI patients and NH listeners [Nava et al., 2014; Schierholz et al., 2015]. The discrepancy between the groups of patients might be caused by the differential use of compensation strategies. Patients with central auditory implants may compensate for inadequate input already in conditions with simple sensory stimuli, while CI patients might use compensatory strategies for more complex stimuli.

Cortical Responses to Modality‐Specific Auditory and Visual Stimulation

The present study explored auditory cortex activity in individuals with different auditory implants during auditory stimulation. Our measurements in free field extend previous findings on the feasibility of measuring AEPs in ABI and AMI patients by means of direct electrical stimulation [Colletti et al., 2007; He et al., 2015; Herrmann et al., 2015]. Our results revealed that the morphology and voltage distribution of AEPs were comparable between NH listeners, CI and ABI patients. However, ABI patients' AEPs and source waveforms showed reduced and prolonged N1 amplitudes when compared with NH listeners. In contrast, N1 amplitudes were not significantly different between CI patients and NH listeners, confirming earlier reports of comparable N1 AEPs in those two groups [Henkin et al., 2014; Schierholz et al., 2015]. This points to an equivalent encoding of the stimulus features in CI patients and NH listeners [Näätänen and Picton, 1987]. Regarding ABI patients, the finding of reduced and delayed N1 peaks suggests experience‐related alterations of auditory cortex functions in patients with central auditory implants. On the other hand, the inadequate input from the implant may cause altered and delayed processing in the auditory cortex of ABI patients. Importantly, the two factors are not mutually exclusive and may jointly mediate alterations of auditory processing in individuals with central auditory implants.

Consistent with the behavioral response patterns, the two AMI patients showed variability in their AEPs. AMI2 revealed a prolonged N1 latency and reduced N1 amplitude, barely approaching the values observed for ABI patients. By contrast, AMI1 revealed N1 latency and amplitude measures similar to those of ABI patients. This seems contradictory to the behavioral results, showing similar RTs in AMI2 and prolonged RTs in AMI1 when compared with the ABI group. However, behavioral performance reflects the combined effects of different sensory and cognitive processes, while the response at N1 latency reflects initial stages of sensory processing. Thus, the poor behavioral performance in AMI1 may result from altered higher level processing rather than from changes in early sensory processing. Moreover, one has to keep in mind that the AMI data presented here are just two single cases and thus should be interpreted with caution.

The lack of differences in VEP latencies and amplitudes between the different groups was consistent with the behavioral results, showing similar hit rates and RTs for the visual condition across all groups (NH, CI, ABI). Furthermore, the results replicate former findings of similar VEPs in response to basic, stationary visual stimuli in young and elderly CI patients and NH listeners [Schierholz et al., 2015]. The present study extents these findings to post‐lingually deafened ABI and AMI patients. A modified visual processing in patients with auditory implants might be observed for more advanced visual stimulation, involving complex patterns or motion [e.g., Sandmann et al., 2012]. Furthermore, the lack of significant differences indicates that only the true visual modulation of the auditory response, and not the visual response per se, was different between the groups.

Cortical Responses to Cross‐Modal Stimulation

We analyzed audio–visual interactions by comparing the unisensory auditory response (A_uni) with the visually modulated auditory response (A_mod; [AV + nostim − V]). A substantial increase in N1 amplitude from unisensory to modulated responses was observed for CI and ABI patients, but not for NH listeners. The visual modulation points to the occurrence of multisensory interactions in CI and ABI patients and it is likely the consequence of auditory deprivation and/or a reduced sound representation with auditory implants. The source analysis revealed that this effect originates, at least to some part, in the auditory cortex. Consistent findings were obtained in our previous study [Schierholz et al., 2015]. The fact that the same pattern of visual modulation was found in both, the CI and the ABI patients, supports the view that auditory deprivation and/or implant use can change audio–visual interactions. Interestingly, only for patients with central auditory implants ‐ but not for the CI patients – the enhanced visual modulation of auditory cortex activation was in line with the behavioral performance. A strong audio–visual gain for speech (Gain_Speech) and non‐speech stimuli (Gain_RT) was observed in the group of ABI patients who revealed a pronounced visual modulation of auditory–cortex activation, indicating multisensory interactions. By contrast, CI patients showed a weak audio–visual benefit on the behavioral level, and a strong visual modulation of AEPs. The inconsistency between CI and ABI patients with regard to the presence of a behavioral correlate may be related to the applied tasks. CI patients in the current study were good performers. Both, the detection of auditory stimuli, as well as the speech tests in silence meant no challenging task for these participants. We speculate that in good CI performers, the enhancement of audio–visual interactions becomes visible on the behavioral level only for more sophisticated stimuli and tasks, as for instance for speech‐in‐noise conditions.

With regard to the AMI patients, AMI1 showed a slight decrease in N1 amplitude from the unisensory to the modulated response. The lack of a (positive) response modulation in this patient was consistent with the lack of a violation of the race model, indicating that the observed gain in performance from unisensory to bimodal stimulation was rather due to statistical facilitation. This may suggest that the pattern of auditory activity in AMI1 was so abnormal that it was too difficult to be fused with the visual response pattern. These challenges may be related to suboptimal placement of the stimulation electrodes and the difficulty in providing excitation patterns to the more central structures that can then be appropriately processed by the brain. Unlike AMI1, AMI2 revealed a strong increase in N1 amplitude from the unisensory to the modulated response, similar to the results of CI and ABI patients. These observations are in line with the findings of a robust Gain_Speech and a violation of the race model (descriptively) for patient AMI2. Thus, the behavioral improvement in AMI2 for the combined audio–visual conditions seems to be—at least partially—based on enhanced audio–visual interactions in the auditory cortex. Likewise, strong audio–visual interactions in the auditory cortex of ABI patients may underlie the remarkable gain in performance when visual and auditory input is combined.

SUMMARY AND CONCLUSIONS

The present study compared hearing abilities and auditory–cortex activation in patients with electrical stimulation at different sites of the auditory pathway. Our results should be interpreted with caution, as they are based on a comparatively small sample size, probably limiting the power of the study.

The results revealed poor auditory performance and AEPs with smaller amplitudes and prolonged latencies in patients with central auditory implants. However, all groups of patients (CI, ABI, AMI) and NH listeners showed facilitated performance in conditions containing redundant (audio–visual) information. Importantly, benefits for audio–visual information were particularly strong in most patients with central auditory implants, for both, speech and non‐speech stimuli. The improved performance for audio–visual conditions seems to be based on enhanced audio–visual interactions in the auditory cortex. We speculate that the enhanced audio–visual interactions are the consequence of a compensatory processing strategy developed by auditory implant patients to overcome the limitations from electrical stimulation. Interestingly, in CI patients the audio–visual benefit was less pronounced when compared with the central auditory implant patients, although the duration of deafness, the implant experience and the age at implantation did not differ across groups. Consequently, the stronger audio–visual benefit in patients with central auditory implants may be related to experience‐related alterations of auditory processing and/or the inadequate auditory input in these individuals. Knowing the detailed impact of the visual modality on auditory processing in implanted patients may provide important information for the optimization of rehabilitation strategies.

Although, the present study focused on the processing in the auditory cortex, structural, and functional changes due to auditory deprivation and CI implantation/rehabilitation are very likely not exclusive to sensory areas. Previous studies also revealed areas like the frontal cortex to show changes in activity related to the auditory status. Findings include both, hearing impaired [Campbell and Sharma, 2013] and auditory implant patients [Berding et al., 2015], and suggest compensation by and resource allocation to frontal regions, when confronted with diminished auditory input.

In general, our results demonstrate that EEG can be a useful tool to evaluate auditory cortex function in patients with auditory implants (CI, ABI, AMI), where imaging methods like magnetic resonance imaging (MRI) are of limited use due to the possible occurring electromagnetic interference. EEG, however, is a non‐invasive, safe technique which might become a standard clinical tool to objectively evaluate the recovery of auditory functions and to monitor rehabilitation programs in patients with CI, ABI, and AMI.