Voice Emotion Perception and Production in Cochlear Implant Users

Abstract

Voice emotion is a fundamental component of human social interaction and social development. Unfortunately, cochlear implant users are often forced to interface with highly degraded prosodic cues as a result of device constraints in extraction, processing, and transmission. As such, individuals with cochlear implants frequently demonstrate significant difficulty in recognizing voice emotions in comparison to their normal hearing counterparts. Cochlear implant-mediated perception and production of voice emotion is an important but relatively understudied area of research. However, a rich understanding of the voice emotion auditory processing offers opportunities to improve upon CI biomedical design and to develop training programs benefiting CI performance. In this review, we will address the issues, current literature, and future directions for improved voice emotion processing in cochlear implant users.

1. Introduction

Communicating emotion is a fundamental feature of human social interaction that transverses all cultures (Bryant & Barrett, 2008). In fact, some may argue that emotional cues formulate the very basis of human interaction and carry more valuable information that the actual words being spoken (Zajonc, 1980). There are many cues that come into play when communicating emotion, one of the most important being nonverbal cues (Skinner, 1935; Wallbott & Scherer, 1986). Among all types of nonverbal cues, humans frequently use prosodic vocal cues (e.g., voice pitch and tempo) to elicit emotive information in their interactions (Planalp, et al., 1996). So naturally, when prosodic vocal cues are degraded, voice emotion perception and production are often affected. Impairments in the perception and production of voice emotion usually result in serious ramifications on social interactions and social development, as in the case of infant-directed speech (Trainor, et al., 2000), underscoring the importance of this topic at hand.

Cochlear implants (CI) are surgically implanted electrical devices that allow people with severe-to-profound hearing loss to process sound. Over the past few decades, CI development has made remarkable ground such that most CI users have adequate speech perception in quiet environments. Despite this great success, limitations remain for present day CI systems including the transmission of spectro-temporal fine structure information (e.g. pitch and harmonics) (Kong, et al., 2004; Galvin, et al., 2007; Kang, et al., 2009; Kong, et al., 2011; Xu, et al., 2009). Forced to interface with highly degraded acoustic cues, CI users often demonstrate difficulty in perceiving prosodic cues. Limitations in the perception and production of prosody have adverse consequences for CI users, including the interpretation and communication of voice emotion. In this article, we will review the emerging body of work on CI-mediated perception and production of voice emotion.

2. Voice Emotion General Principles

2.1 Dimensions of Emotions in Relation to Speech Emotion Studies

Emotions are brief and strong reactions to goal-relevant changes in the environment. Historically, there are two main approaches towards studying emotion: discrete and dimensional. A discrete approach focuses on characteristics that distinguish emotional states from one another (Ekman, 1992) whereas a dimensional approach identifies emotions based on predetermined features underlying mood and affective states (Russell, 1980). Although there are many dimensions involved in emotion, the four most commonly referred-to dimensions of subjective feeling states are activation, valence, potency, and intensity (Smith & Ellsworth, 1985). Activation refers to the perceived sense of energy ranging from low to high (e.g. somnolence to feverish excitement) (Krumshansl, 1997; Gosselin, et al., 2007; Sammler, et al., 2007). Orthogonally, valence relates to the intrinsic evaluation of an event, object, or situation and ranges from positive to negative (e.g. joy to displeasure) (Krumhansl, 1997; Schubert, 1999; Dalla Bella et al., 2001). Potency is a dimension used to describe the degree of powerfulness or powerlessness an individual universally identifies with a particularly emotion (Russell & Mehrabian, 1977; Osgood, et al., 1957). Positive emotions almost always generate a high level of control or dominance. Thus, potency is specifically useful in differentiating between negative emotions such as fear and anger; where anger has a high potency rating and fear has a low potency rating. Last but not least, emotional intensity is used to quantify the degree of emotion being felt (e.g. very happy or only a little bit happy). The valence-arousal model approaches emotion as two separable dimensions of valence and arousal (Russell, 1980). These two dimensions are commonly used in vocal expression studies (Bachorowski, 1999, Scherer, 1986) and capture the majority of the psychophysiological components of emotion, which is why some researchers reduce emotion theory down to only two components: valence and arousal.

2.2 Speech Prosody Cues & Voice Emotion

The origin of the term ‘prosody’ can be traced back to ancient Greek where it was used to indicate the tone or accent of a syllable. Over the years, the word prosody has evolved to govern the modulation of the human voice when uttering segmental sequences of phonemes. In modern phonology, prosody refers to elements of speech relating to the properties of syllables and larger units of linguistics, such as voice pitch, duration, intensity, spectral characteristics, nonverbal vocal expressions (e.g. crying), rhythm, and tempo. Prosodic features of speech often cannot be captured by conventional segmental phonetic transcriptions or orthography. These properties of speech play an important role in communication, such as informing a listener of the speaker’s intent and affect. Overall, there are few articles on the acoustic correlates of emotion. Below, we chose to highlight the most common associations found between prosodic cues and voice emotion activation and valence. The general principles mentioned in this section are not steadfast rules particularly with findings concerning valence.

Strictly speaking, pitch is the perceptual correlate of the fundamental frequency (F₀) of a sound. Although pitch is a subjective attribute and fundamental frequency an objective acoustical parameter, the two terms are often used interchangeably in the literature. For the purposes of this review, we will also use the word ‘pitch’ to refer to F₀. In speech, the fundamental frequency is derived by the rate of vocal cord vibration. The fundamental frequency range varies between speakers and depends greatly on the length and mass of the vocal cords – As a result, male speakers (85 to 180 Hz) generally have a lower fundamental frequency range than female speakers (160 to 255 Hz). Within their individual ranges, speakers have a large degree of active control over voice pitch and can choose to speak in a high or low pitch with corresponding rises and falls.

As previously mentioned, pitch is a strong acoustic cue for voice emotion in both children and adults. In fact, many school-aged children perform to the same level as adults in pitch discrimination tasks, demonstrating that fine fundamental frequency cues in voice can be available at a young age (Figure 1) (Deroche, et al., 2012). It is yet unclear whether the ability of children to recognize emotion in voice develops because this fine sensitivity to pitch is available to them very early on, or on the contrary whether voice emotion processing is one of the causes driving the auditory system to refine its sensitivity to pitch. But it is clear that the two aspects are tightly connected. For example, high activation is often associated with a high mean fundamental frequency (Breitenstein, et al., 2001; Davitz, 1964; Levin & Lord, 1975; Pereira, 2000; Scherer & Oshinsky, 1977; Schröder, et al., 2001) and fundamental frequency variability (Breitenstein, et al., 2001; Pereira, 2000; Scherer & Oshinsky, 1977). Studies involving valence and voice pitch, on the other hand, are much less consistent. Some authors observe positive valence with low mean fundamental frequencies and high levels of fundamental frequency variability (Scherer & Oshinsky, 1977; Uldall, 1960) whereas other investigators fail to find patterns of vocal cues for valence dimension (Apple, et al., 1979; Davitz, 1964; Pereira, 2000).

Figure 1.

17 school-aged children were asked to discriminate sinusoidal amplitude modulation rate (AMR) of broadband noise and to discriminate the fundamental frequency (F0) of broadband sine-phase harmonic complexes (Deroche, et al., 2012). Children who had adult-like sensitivity were not necessarily the oldest listeners. For example, the youngest research subject was 6.5 years old and demonstrated a standardized threshold of only 1.08 semitones for AMR discrimination and 12 cents for F0 discrimination, suggesting that children’s sensitivity to pitch (regardless of the underlying cue) does not systematically improve beyond 6 years of age.

With most natural sounds, duration is determined by the time interval between an onset and offset. This is not only applicable for duration of sounds, but also for duration of silence between two sounds. In cases of clear and rapid changes in the sound stimuli, perception of the phonemic segmentation is more or less straightforward. With slower changes in the durations of sound segments or silence intervals, such as in glides and slurred speech, speech becomes subject to listener perception and interpretation. In general, shorter pauses are associated with high activation (Schröder, et al., 2001). Longer pauses are commonly observed with negative valence (Schröder, et al., 2001).

Tempo is a common acoustic cue used to convey emotion. Emotions with high levels of activation (e.g. excitement and anger) and positive valence are commonly associated with fast speech rates (Apple, et al., 1979; Breitenstein, et al., 2001; Davitz, 1964; Kehrein, 2002; Scherer & Oshinsky, 1977).

Vocal intensity is the sound pressure level (Isshiki, 1964) and is an important prosodic cue used in voice emotion. In general, high-frequency energy is highly predictive of perceived intensity in listeners (Juslin & Laukka, 2001). Previous studies frequently report increased voice intensity with high activation and negative valence (Davitz, 1964; Huttar, 1968; Pereira, 2000; Schröder, et al., 2001). However, vocal intensity is relatively overlooked in relation to voice emotion, and the relationship between intensity and other emotion dimensions is not well understood.

It is important to bear these associations in mind, particularly in the light of recent interest into speaking styles such as “clear speech”. To facilitate conversations with listeners presenting hearing losses, speakers may raise their voice’s intensity as observed in the Lombard effect (Lane & Tranel, 1971), may speak at a slower rate and mark longer pauses between words. Although this may provide notable benefits in terms of intelligibility, e.g. better signal-to-noise ratio and enhanced syllable segmentation, this speaking style is likely to carry more negative valence and in fact, recent evidence suggests that this is exactly the case (Morgan & Ferguson, 2014). Therefore, beyond audiologic considerations alone, there is a need for a better understanding of the emotions associated with different manipulations of prosodic cues in speech.

3. Review of Prosody and Voice Emotion Studies in Cochlear Implant Recipients

While speech recognition in CI users is relatively well studied, speech prosody and emotion are less so. Salient voice pitch information and pitch-based harmonic structures are demonstrably important components of speech prosodic cues. However, present day CI systems provide limited spectro-temporal fine structure information in speech (Geurt & Wouters, 2001; Green, et al., 2004). This in turn impairs CI users’ ability to perceive and produce important prosodic forms of communication such as question-statement contrasts, lexical tone recognition, and voice emotion (Shannon, 1983; Zeng, 2002; Chatterjee & Peng, 2008; Luo, et al., 2007; Peng, et al., 2004; Luo & Fu, 2004; Ciocca, et al., 2002; Wei, et al., 2004). Because voice emotion perception relies on many overlapping prosodic cues, the general consensus is that adults and children with CIs have significant difficulty in interpreting emotions. Critically, however, studies indicate wide inter-individual variability among CI users in emotion- and prosody- related tasks. In this section of this article, we will review the body of literature on CI171 mediated production and perception of prosody and voice emotion.

3.1 Voice Emotion and Prosody Perception

Many of the studies investigating CI users’ perception of emotion in speech use professional actors’ recordings of semantically neutral sentences (for example, “The coat is on the chair”) to acoustically convey targeted emotions such as happy, sad, angry, anxious, relieved, or scared. Studies like these consistently observe significant deficits in target emotion recognition in CI users, both adults (Luo, et al., 2007; Pereira, 2000; Kalathottukaren, et al., 2015) and children (Hopyan-Misakyan, et al., 2009; Chatterjee, et al., 2015; Nakata, et al., 2012; Volkova, et al., 2012). Gilbers et al. (2015) also demonstrated poorer performance in CI users than NH listeners in an emotion identification task using nonce words. NH listeners performed better than CI users even when presented with CI simulations of stimuli.

In general, children and adult CI users can differentiate questions from statements, especially when envelope periodicity cues are used (Rosen, 1992). As a group, CI users’ overall performance is well below normal hearing listeners (Peng, et al., 2008; Most & Peled, 2007; Van Zyl & Hanekom, 2013; Chatterjee & Peng, 2008; Meister, et al., 2009; Peng, et al., 2012) and severely-to-profoundly deaf pediatric hearing aid users (Most & Peled 2007). Statements typically end with a decrease in pitch relative to the remainder of the sentence as compared to an increase in pitch at the end of questions. The low CI performance on statement-question discrimination tasks is likely due to the poor representation of the pitch contour in CI users (See, Driscoll, Gfeller, Oleson, & Kliethermes, 2013). Crucially, this limitation can also affect representation of lexical tone languages, such as Mandarin, which is based on differences in the height and movement of pitch on a vowel phoneme. CI-mediated peer-to-peer interactions are often inhibited in lexical tonal languages because present day CI systems have difficulty detecting and transmitting rapid intonation changes within the syllable. This handicap ultimately stems from degraded pitch cues, and studies have observed significant impairment in identifying and producing lexical tones among CI users (He, et al., 2016, Peng, et al., 2004; Ciocca, et al., 2002; Holt & McDermott, 2013; Deroche, et al., 2016; Deroche, et al., 2014; Wang, et al., 2011; Wong & Wong, 2004).

Emphasis, or stress patterns in speech, is also important to vocal emotion communication and is dependent on the same prosodic cues. Among other findings, Meister and colleagues (2009) reported that CI users performed more poorly than NH listeners when asked to identify the stressed word (which varied between subject, verb, and object) in a series of sentences. The authors demonstrated similar results using stimuli in which the speakers were instructed to stress a particular word and stimuli incorporating pitch manipulation to artificially stress or de-stress the word. This suggests that CI users are not particularly effective at using alternative strategies based on intensity and duration cues that are available in naturally uttered stresses. In other words, they rely heavily on pitch information despite its poor quality. Kalathottukaren et al. (2015) also found impaired stress identification ability in CI users using the Profiling Elements of Prosody in Speech-Communication (PEPS-C) test, a battery of four prosody tests measuring perception of contour, affect, stress, and “chunking”, referring to utilization of prosodic cues to communicate intonational subsections of a phrase (for example, “fruit salad” as a compound word versus “fruit, salad” as a list). CI users performed more poorly than NH listeners in the prosodic subtests dependent on pitch representation (the contour, affect, and stress tests) but near normal on the “chunking” test, which relies more heavily on temporal cues. This is another manifestation of poor pitch perception in CI recipients but also suggests that they are able to utilize durational and timing cues as one means of extracting prosodic information (Figure 2, Kalathottukaren, et al., 2015).

Figure 2.

Percent correct scores for CI users (left) and NH listeners (right) on four subtests of PEPS-C, including turn-end, affect, chunking, and contrastive stress (Kalathottukaren, et al., 2015). As indicated by the asterisk, chunking was the only task for which CI users did not perform significantly worse than NH listeners.

Speech emotion perception in pediatric CI patients is of particular interest among researchers because the developmental implications of early auditory deprivation could have notable long-term effects; for example, infant-directed speech uses exaggerated prosodic cues to convey emotion, which help the infant understand the intended emotion of the adult and aid in development of communication skills (Trainor, et al., 2000; Soderstrom, et al., 2003). Early exposure to speech and prosody could play a significant role in the social development of CI children. The results of a study by Wiefferink et al. (2013) suggest that pre-school-aged children with CIs are delayed in facial affect recognition and emotional attribution abilities (Wiefferink, et al, 2013). This may be surprising given that facial emotion was conveyed visually to CI users who had normal vision and in fact, one might have expected these users to have learned to make a better use of visual cues to compensate for their auditory deficits. Interestingly, by school age, CI children seem to have established normal facial emotion recognition (Hopyan-Misakyan, et al., 2009). This finding suggests that, in the first few years of brain development, the connectivity between auditory information and emotion helps conceptualize human emotions. The loss of one emotion processing modality presumably leads to concepts that are less-well-defined which may then have repercussions on the emotional connectivity with the visual system itself. Ketelaar et al. (2013) measured empathic behaviors, emotion acknowledgement, and social competence in CI and NH children and found that, in contrast with other studies (Weifferink, et al., 2013), social competence and empathic behaviors were not impaired in the CI group. The authors also found that emotional acknowledgement predicted social competence in the CI group but not in the NH group, possibly suggesting that CI children are more attentive to visual emotion signals, perhaps to compensate for a diminished auditory input. Notably, the study reported language scores in their CI group higher than previously reported averages for CI children implanted prior to age two (Boons, et al, 2012). Furthermore, cognitive abilities were not measured in either group. It is therefore difficult to pinpoint the source of these findings.

Research investigating music emotion processing in CI users further supports the premise that CI users utilize alternative processes to process auditory emotion (Volkova, et al., 2012; Hopyan, et al., 2011; Shirvani, et al., 2015; Giannantonio, et al., 2015). Hopyan et al (2011) found that CI children performed significantly more poorly than their NH peers in distinguishing happy vs. sad music. The authors hypothesize that the CI group utilized tempo rather than pitch cues, which are fundamental to musical emotion, in making this distinction. Indeed, Caldwell et al. (2015) found that adult CI users rely on tempo rather than pitch in the processing of musical emotion, whereas NH listeners relied on both cues. The results of a study by Giannantonio et al. (2015) parallel these findings in children, additionally indicating that CI children’s reaction times in musical emotion identification were affected more strongly by changes in tempo compared to mode, whereas those of NH children changed more significantly with concurrent changes in mode and tempo. These studies strongly support the notion of a unique processing strategy for emotion in auditory cues – in particular, by increased reliance on tempo-based modalities – in CI users compared to NH listeners.

More studies including adult populations are required to shed light on auditory input and long-term brain development in terms of emotion recognition (we will discuss the cortical reorganization effects of auditory deprivation later in this article). In a recent study (Chatterjee, et al., 2015), pediatric CI users performed similarly to adult CI users on tasks of speech emotion perception, and both performed comparably to NH adults listening to vocoded speech. However, children with NH performed more poorly than NH adults in these CI simulations (Figure 3, Chatterjee, et al., 2015). This could indicate that adults, both CI and NH, have sufficiently developed cognitive systems to be able to decipher emotion from degraded speech, while CI children may have found a way to adapt to the impoverished auditory signal. NH children listening to vocoded speech have neither fully developed cognitive systems nor learned adaptive methods of emotion, potentially explaining why they have trouble with deciphering voice emotion from CI simulations.

Figure 3.

Mean emotion recognition scores for adult NH, adult CI, child NH, and child CI study groups across full spectrum speech and speech presented with three levels of spectral degradation (Chatterjee, et al., 2015). Child and adult CI users’ performance were similar to one another’s and comparable to adult NH’s performance in 8-channel degraded speech.

Acoustic analysis of questions/statements recordings by NH listeners demonstrates the importance of pitch contour, and to a much lesser degree intensity and duration (Peng, et al., 2012). When presented with many versions of the same contrasts manipulated incrementally along one dimension, NH listeners rely heavily on pitch contour. In contrast, listeners with poor spectral information (e.g. CI users or NH listeners attending to noise-vocoded stimuli) tend to modify their listening strategies to rely more on secondary cues such as intensity and duration (Figure 4, Peng, et al., 2012). As demonstrated, when F0 cues are degraded as it is the case in CI processing, other prosodic cues may be used to extrapolate emotion information, such as intensity and duration, although they are not as reliable (Pereira, et al., 2000; Luo, et al., 2007; Gilbers, et al., 2015). For example, intensity normalization has a greater impact on CI users’ performance in emotion recognition than NH listeners’. This is consistent with the idea that CI users rely relatively more on intensity cues compared with NH listeners (Luo, et al., 2007). Duration may also be important to CI recipients for accessing emotion information in the absence of pitch and/or amplitude cues (Hegarty & Faulkner, 2013). Finally, CI users may utilize the limited pitch cues they have, but in a different manner than NH listeners. For example, Gilbers and colleagues (2015) asked speakers to produce salient nonce words. They analyzed the acoustic features of the recordings in conjunction with emotion identification performance by CI users and NH listeners. They determined that mean pitch was more important to NH listeners, while CI users weighed pitch range more heavily. This could indicate that the impoverished acoustic signal in CI users leads to a lack of understanding of the relationship between mean pitch and emotion (for example, the association of high frequencies with high activation) and thus they tend to rely on large pitch differences and patterns to extrapolate emotional information.

Figure 4.

Cochlear implant (CI) versus normal hearing (NH) group mean proportions of question judgements as a function of an acoustic dimension (F₀ height, F₀ contour, peak intensity ratio, and duration ratio) (Peng, et al., 2012). The NH group was also subject to different listening conditions (full-spectrum, 8-, and 4-channel conditions) as depicted by different symbols.

3.2 Voice Emotion and Prosody Production

Few studies have examined voice emotion production in CI users. Prosody production in CI users is more studied in children than adults, though even this research has been sparse. Acoustic voice quality in CI children is generally better than that of hearing aid users (Guerrero Lopez, et al., 2013); however, there remain significant impairments in emotion production in the CI population. Studies report less accurate imitations of happy- and sad-sounding speech among CI children compared to NH children, particularly with regard to appropriate pitch modulation (Wang, et al., 2013). Furthermore, CI children demonstrate difficulty with perceiving more subtle emotions such as disappointment and surprise (Nakata, et al., 2012). Speech production studies suggest a lack of appropriate pitch contour in conveying questions and/or statements in CI children (Peng, et al., 2008; Peng, et al., 2007; Chin, et al., 2012). Recent preliminary data from Chatterjee et al. (2016) illustrates that CI children’s productions of emotional speech incorporate smaller contrasts between happy and sad speech in mean pitch, pitch range, intensity, and spectral centroid compared to NH children. Post-lingually deafened CI adults’ speech exhibited acoustic patterns more similar to the NH population.

3.3 Factors Influencing CI-Mediated Voice Emotion

CI-mediated voice emotion abilities are not uniform across the CI population. There is large inter-individual variability in speech prosody perception and production skills. In fact, many studies report a handful of “star” CI performers whose abilities are reportedly on par with their NH peers (Wang, et al., 2013; Chatterjee, et al., 2015). A myriad of factors contributes to variability in speech prosody skills including auditory deprivation prior to implantation, age at implantation, and technological factors.

As discussed above, the role of early CI implantation and auditory deprivation in appropriate development of speech emotion skills is highly complex and not well understood. Early implantation is consistently correlated to better performance in speech intelligibility tasks in congenitally and pre-lingually deaf children, demonstrating the importance of early auditory input in proper emotional speech development (Schorr, et al., 2009; Artières, et al., 2009). This is reinforced by neurophysiological evidence. Neurological differences between deaf and hearing people are well established, with functional reorganization occurring as a result of decreased auditory input. Electrophysiological studies of deaf children with and without CIs at specific periods of development suggest that a lack of auditory stimulation hinders auditory cortex development. CI usage can stimulate maturation at a rate close to normal, at least in parts of the auditory pathway; but this is limited by the degree of reorganization that occurred during the period of deafness (Ponton, et al., 2000; see Gordon, et al., 2011 for review).

Variations in technology can also influence CI users’ prosody perception. Bimodal CI users, i.e. those using a CI and contralateral hearing aid simultaneously, are better at identifying intended emotion than deaf individuals using CIs alone (Straatman, et al, 2010). Processing strategies play a significant role in voice emotion as evidenced by both behavioral and electrophysiological data. Agrawal et al. (2013) used EEG to compare speech emotion perception and even-related potential (ERP) signals in NH listeners, CI recipients using the Psychoacoustic Advanced Combination Encoder strategy (also known as MP3000), and CI recipients using the Advance Combination Encoder strategy (i.e. ACE). M3000 functions to more precisely transmit spectral information by maximizing representation of relevant spectral information using psychoacoustic masking, whereas ACE focuses less on eliminating redundancies and instead on transmitting acoustically salient information (Wouters, et al., 2015). NH listeners’ performance on the sentence emotion identification task was significantly more accurate than CI users, and their P200 ERP had a significantly higher amplitude than that of the CI group. CI users using MP3000 demonstrated more accurate identification of emotion in spoken sentences than those using ACE, and a more positive P200 ERP response (closer to NH performance) when assessing “happy” prosodic cues (Figure 5, Agrawal, et al., 2013). These results demonstrate the potential impacts of CI processing strategy parameters on emotion perception, and in particular that prosody processing might be optimized by strategies that target a focused representation of spectral information. In a similar study, Agrawal et al (2012) observed improved prosody perception when using MP3000 rather than standard CI simulations among NH listeners.

Figure 5.

Accuracy rate in identification of neutral, angry, and happy prosody in NH listeners and CI users using ACE and MP3000 strategies (Agrawal, et al., 2013).

3.4 Rehabilitation in CI-Mediated Voice Emotion Perception

There is evidence that aural rehabilitation may improve CI-mediated detection of suprasegmental features of speech, such as intonation, pitch, intensity, and duration. A Chinese rehabilitation program with 28 prelingually deaf pediatric CI users demonstrated improved sentence recognition and story comprehension after two years of training (Wei, et al., 2000). Similarly, a pilot study implemented an 11-lesson psychoeducational program focused on improving emotional understanding among 14 deaf children and observed significant increases in emotion vocabulary and emotion comprehension from pretest to posttest. (Dyck & Denver, 2003). Although these findings imply that emotion recognition ability may be improved with training, the emotion training program heavily focused on using facial emotion cues and less so on vocal emotion cues. On the other hand, Zhang et al. (2013) did not find improved performance with voice gender and emotion identification tasks among 7 unilateral CI and contralateral acoustic stimulation users after 4 weeks of training, totaling 20 hours, suggesting that any improvement in emotion processing would require intense training over a sustained period of time.

Musical training is an increasingly popular avenue for speech emotion rehabilitation in CI users, as evidenced by studies indicating improvements in prosody perception as a result of musical engagement, training, or exposure. Longitudinal musical training studies consistently suggest that music training provides speech-related benefits, such as phonological awareness, perception of vowel duration, and speech segmentation, to NH listeners (Hausen, et al., 2013; Degé & Schwarzer, 2011) and benefits in melodic and pitch perception to CI users (Chen et al, 2010; Galvin, et al, 2007). Torppa et al. (2014) found that CI children participating in music or dance activities over a period of 16 months performed similarly to NH children in tasks of pitch discrimination and word stress perception, whereas CI children participating in non-musical tasks over the course of the study performed more poorly. Patel (2014) provided preliminary evidence that melodic contour training with an emphasis on contour precision perception can improve intonation perception, and a recent study (Yhun Lo, et al., 2015) demonstrated improvements in question-statement discrimination and other speech perception parameters as a result of melodic contour training in adult CI users, both interval-based and duration-based (Figure 6, Yhun Lo, et al., 2015). These studies suggest that auditory training may serve as a rehabilitation mechanism for speech emotion perception skills in CI users. Nonetheless, the number of aural rehabilitation studies are meager and more investigation in this field of work is needed before conclusions can be drawn regarding the benefits of auditory training on voice emotion recognition.

Figure 6.

Graphical representation of improvements in question-statement discrimination (as measured by the PEPS-C test) in CI users after participation in melodic contour training (Yhun Lo, et al., 2015). Two programs of melodic contour training were administered. The “Interval” program incorporated manipulations of note intervals, whereas the “Duration” program changed the note durations, in order to make the tests more challenging. The dashed line represents chance performance.

3.5 Conceptualization of Voice Emotion and Musical Terminology in CI Users

It is important to note the difficulty CI users and children with NH face in conceptualizing auditory terms. Young children with NH often use terms associated with loudness to describe changes in pitch (Andrews & Diehl, 1970; Hair, 1981; Van Zee, 1976). In English, the words used to describe the pitch scale, “high” and “low”, also have spatial, emotional, and loudness connotations. However, in other languages (e.g. Spanish and French) descriptor terms for pitch are specific and used only in reference to pitch, and this language factor alone results in an improved ability to label the direction of pitch change (Costa-Giomi & Descombes, 1996). This phenomenon is important to acknowledge because it suggests that a given subject may demonstrate deficits in a task simply because he or she did not fully understand the concept of the auditory cue within the research task.

In the case of CI users, despite scientists’ best efforts to explain auditory terms in a simplistic but accurate manner, if the representation of the cue is poor (as in the case of fundamental frequencies), CI users’ understanding of it may be extremely vague. This could be especially problematic with pre-lingually deaf CI subjects whom may have little reference and exposure to musical terms and sounds. In a sense, this effect could potentially confound study findings and thus, our understanding of CI-mediated processing because subjects cannot perform well in a task that they don’t understand. These events may frequently occur in pitch and voice emotion; for example, a CI user may base his or her concepts of emotion on facial expressions and his or her own set of experiences and biases given a lack of dependability on impoverished auditory cues. However, when these visual cues are removed in a research experiment and the CI subject is asked to identify the voice emotion of a sound stimulus, a CI users’ understanding of the task at hand may be akin to a NH subject being asked to use auditory cues to discriminate between a green sound and a blue sound. This analogy attempts to describe how the perception of emotion in music and speech involves cues that CI users do not have good access to and may not naturally associate with voice emotion. On the other hand, we recognize that despite these concerns of conceptualization, it may be necessary to use cues of interest within the auditory modality given that restoration of hearing is the ultimate goal of cochlear implantation. Nonetheless, this conceptualization issue should be acknowledged and may be addressed with careful experimental implementation, standardization in descriptor terms for music, and music training, depending on a listener’s ability to internalize and then verbalize these concepts, and later create personal cognitive meanings to them.

Although we emphasize the unique nuances of limited familiarity with voice emotion and musical terminology with respect to tasks involving emotion, there are many other factors that could confound study findings. Indeed, research subjects are often tested in study conditions that are not typical for them. For instance, bilateral listeners are often tested using only unilateral input, NH listeners interface with vocoded sounds, CI users grapple with new processors and programs, and test batteries involve unfamiliar and often meaningless stimuli. All of these conditions may confound study results, however, we would argue that these inherent constraints in study design may not be as specific to experiments involving emotion in speech and music. All in all, the significance of these issues involving unfamiliar study conditions remains of importance and may be more easily addressed than poor conceptualization secondary to limited exposure to emotional information.

4. Conclusion

The role of voice emotion perception and production in communication cannot be overstated. As evidenced by a combination of behavioral and electrophysiological data, CI users face significant deficits in prosodic cues due to significant limitations in pitch, intonation, and contour perception. Our review discusses how voice emotion processing, perception, and production may be hindered in CI users, and that the wide variability in CI performance is secondary to a number of factors including poor pitch representation and limited spectro436 temporal fine structure information. As a result, CI users may be forced to rely on cues other than frequency - such as intensity, tempo, and duration - to determine the intended emotion of a speaker (Kalathottukaren, et al., 2015). Other studies have shown that CI users are not particularly effective at using alternative strategies, and continue to rely most heavily on pitch information despite its poor quality for prosodic information (Meister et al., 2009).

Adult and pediatric CI users consistently demonstrate significant deficits in target emotion recognition ((Luo, et al., 2007; Pereira, 2000; Kalathottukaren, et al., 2015; Hopyan-Misakyan, et al., 2009; Chatterjee, et al., 2015; Nakata, et al., 2012; Volkova, et al., 2012; Gilbers, et al., 2015) and other prosodic-dependent components of communication (e.g. question-statement discrimination, lexical tone languages, stress patterns in speech). Voice emotion perception is of particular interest to researchers because of its implications on social interactions and auditory development in children. With the increasing population of implanted children and adult, our understanding of the impact of early auditory deprivation, CI use, and neuroplasticity in speech emotion development is improving. These are mixed findings regarding pediatric CI users and their competence in areas of emotion perception including facial affect recognition, empathetic behaviors, and emotional attribution abilities. However, the literature on this topic is severely limited and emotion processing in children using CIs is not well understood. Both adults and children with CIs, however, seem to perform comparably with NH adults in CI simulations, suggesting that this population of people may have sufficiently developed adaptive strategies to decipher emotion from impoverished auditory signals. Among all the auditory cues, pitch cues appear to be the most poorly represented auditory stimulus among CI users for vocal emotion identification and production, and a focus on pitch perception rehabilitation and sound processing may improve performance in CI users.

Highlights.

• Voice emotion is a fundamental component of human social interaction and development.

• Cochlear implant (CI) users are forced to interface with highly degraded prosodic cues.

• CI users demonstrate significant difficulty in recognizing and producing voice emotions.

• Aural rehabilitation and music training may improve prosody perception.