Facial Expressions as an Index of Listening Difficulty and Emotional Response

Abstract

Knowledge about listening difficulty experienced during a task can be used to better understand speech perception processes, to guide amplification outcomes, and can be used by individuals to decide whether to participate in communication. Another factor affecting these decisions is individuals' emotional response which has not been measured objectively previously. In this study, we describe a novel method of measuring listening difficulty and affect of individuals in adverse listening situations using automatic facial expression algorithm. The purpose of our study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We recorded speech recognition scores, facial expressions, subjective listening effort scores, and subjective emotional responses in 33 young participants with normal hearing. We used the signal-to-noise ratios of −1, +2, and +5 dB SNR and quiet conditions to vary the difficulty level. We found that facial expression of confusion and frustration increased with increase in difficulty level, but not with change in each level. We also found a relationship between facial expressions and both subjective emotion ratings and subjective listening effort. Emotional responses in the form of facial expressions show promise as a measure of affect and listening difficulty. Further research is needed to determine the specific contribution of affect to communication in challenging listening environments.

Listening Effort and Its Measurement

Listening effort can be defined as the “deliberate allocation of mental resources to overcome obstacles in goal pursuit when carrying out a task.” ¹ Measuring listening effort can provide more information than is obtained in traditional speech recognition tests. Though it is tempting to consider speech intelligibility scores as a proxy for listening effort, this is not always true. ^{2 3} Winn and Teece ³ showed that in the presence of mispronounced or incorrect words, even when participants give correct responses, they show increased listening effort possibly due to the effort spent in decoding the words. Hence, listening effort provides more information than the speech intelligibility results alone. Listening effort can also be used to demonstrate effectiveness of hearing aid features such as noise reduction. Wendt et al ⁴ measured speech intelligibility and listening effort near ceiling performance (i.e., at 95% correct SNR level with and without noise reduction activated in hearing aids). They found that though speech intelligibility performance was similar for both these conditions, peak pupil dilation was reduced when noise reduction was activated, illustrating the usefulness of this feature and also the usefulness of measuring listening effort in addition to speech intelligibility. Similar results of reduced listening effort with noise reduction activated, despite similar word recognition scores, was also found using of dual-task paradigm to measure listening effort. ⁵ Furthermore, measuring listening effort also gives us information about whether individuals with hearing loss are inclined to participate in different communication situations. Individuals with hearing loss mentally weigh the listening effort cost to the social connectedness benefit when evaluating participation in a communication situation. ⁶ The aforementioned studies show how listening effort provides additional information for speech perception, amplification benefit, and social-participation cost in communication situations. Thus, there is merit in studying listening effort to understand the speech perception process better as well as to guide rehabilitation processes. Another factor that may affect individuals' participation in communication situations, and their use of amplification, is their emotional responses to communication challenges. ⁷ Listening effort and affect, especially in a challenging listening situation, may be intertwined ultimately impacting the communication situation. Previous studies have not focused on emotions present in communication in adverse listening situations. Since emotional responses could also affect the communication process, and this is relatively less studied, it may be important to assess both listening effort and affect.

Various measures have been used to study listening effort. They can be categorized into behavioral measures such as dual-task paradigm; physiological measures such as pupillometry, heart-rate variability, and skin conductance; and subjective measures such as rating scales. Dual-task paradigm involves performing two tasks simultaneously: a primary and a secondary task. When measuring listening effort, the primary task is speech perception, while the secondary task could be a word recall task or a visual/verbal reaction time task. ⁸ This paradigm works on the rationale that individuals have limited cognitive resources. When difficulty for the primary task increases and the listener tries to maintain their performance for the primary task, due to limited resources, this affects their performance on the secondary task which is a measure of the listening effort. Dual-task measures are reliable in measuring listening effort and consistently show worse performance in the secondary task with increasing difficulty of the primary task. ^{5 8 9 10} Some concerns raised about this method are that there is no way to ascertain that listeners are devoting all their resources to the task and that performing two tasks might affect the scores of both. ¹¹ Furthermore, they often require as many as 60 practice trials or more before the subject can do the test. ⁸ This procedure also requires that individuals have the cognitive ability to understand all the instructions. Hence, an easier test of listening effort might be better for clinical implementation.

Physiological measures of listening effort gage the state of the parasympathetic system which is activated with an increase in stress. ¹² Thus, it is assumed that this is activated during challenging listening tasks. One example of a physiological measure is pupillometry, which has been validated as a measure of listening effort. ^{4 13} Pupillometry in this context involves monitoring the pupil size during difficult listening situations. Increased pupil dilation is an indication of increased listening effort. Some drawbacks with using pupillometry are the requirement of additional equipment, training, as well as certain controlled situations (e.g., appropriate lighting). Other psychophysiological measures have also been studied to measure listening effort. Heart rate variability measures also work on the same principle of arousal of the sympathetic/parasympathetic nervous systems. Heart rate variability shows change with hearing loss, ¹⁴ and with change in task difficulty. ^{15 16} However, it is also seen that the changes associated with heart rate variability require larger changes in task difficulty for corresponding changes in these measures ^{14 15} than are required for other measures of objective listening effort such as pupil responses. ¹⁷ Skin conductance, which is a measure of the amount of moisture excreted from eccrine glands, is also a measure of the sympathetic nervous system activity. Studies show a higher skin conductance in speech recognition tasks with the presence of hearing loss, ¹⁴ with increases in task difficulty, ^{14 16} and with changes in task complexity. ¹⁵ Though skin conductance showed increases from baseline for speech in noise tasks, it was not found to be sensitive to changes across signal-to-noise ratios (SNRs). ^{12 15} Hence, psychophysiological measures require extra instrumentation, certain environmental conditions for its implementation, and their validity as listening effort measures is still under investigation.

Subjective measures of listening effort include subjective rating scales. In rating scales, participants report their perceived listening effort after every block of stimuli or after a condition. These are advantageous because they provide us a measure of listening effort perceived by the listener. However, rating scales require internal consistency and validation that individuals can successfully rate their perceived effort rather than their performance. ^{11 18}

Recently, listening effort has been viewed as a multidimensional construct, with “listening effort” as an umbrella term for multiple phenomena and the different measures of listening effort tapping into its multiple underlying dimensions. ¹⁹ Further corroboration of this finding has been found in other studies ^{9 20} that show that results obtained using different measures of listening effort have no association with each other. Objective measures of listening effort such as pupil response or other physiological measures are not found to correlate with subjective listening effort ratings. ^{14 15 16 21 22 23} Thus, based on the construct of interest, we could choose different measures of listening effort. In the current study, we are proposing a new measure that can assess listening difficulty as well as emotional responses using facial expressions. This test will help us determine the affect component present in adverse listening situations along with the listening difficulty.

Emotions and Facial Expressions

Emotions are “complex reaction patterns, involving experiential, behavioral, and physiological elements, by which an individual attempts to deal with a personally significant matter or event” (Dictionary of the American Psychological Association, 2019). Emotions help us communicate our internal feelings to others and help us connect to humanity. Emotions also change our physiology, cognition, and behavior in many ways. ²⁴ Positive emotions help promote engagement with the world around us, ^{25 26} while negative emotions help resolve a particular problem or increase attention toward an activity. ²⁴ Emotional response is the response of an individual to emotional stimuli, for instance, feeling happy after watching a funny clip. Emotional responses are expressed either verbally (pitch, tone, prosody) or through nonverbal communication (facial expressions, body language, gestures).

Relatively less is known about emotional response in individuals with hearing loss. The few studies that investigate the hearing-impaired listener's emotional response show a relationship between emotional responses to non-speech sounds and feelings of social disconnectedness, ²⁷ and also that these individuals have a reduced range of emotional responses to non-speech sounds. ²⁸ For normal emotional processing, individuals need to experience a full range of positive and negative emotions. This is particularly important for individuals with hearing loss because they may already face some degree of social isolation due to hearing loss, which may be aggravated if their emotional processing is inadequate. It is also important to investigate emotional responses to speech stimuli since speech stimuli is of utmost importance for communication. Existing studies in emotional response have not focused on emotional responses to speech. Moreover, existing studies on emotional responses in hearing loss have used subjective rating scales to assess emotional response. ^{27 28} Self-reporting emotional responses are problematic ²⁹ because individuals face difficulty identifying their own emotions, ³⁰ may hesitate to give their honest emotional ratings, ³¹ and may provide ratings that may be rationalizations based on their evaluation of the stimuli under consideration. ³² Hence, an objective measure may prove to be a better measure of emotional responses. Currently, there are no established objective measures to study emotional responses in individuals with hearing loss. Existing literature in psychology reveals mixed findings regarding the relationship between subjective and objective emotional responses. ^{33 34 35 36 37} Matsumoto et al ³⁸ suggested that a link between objective emotions and subjective experiences exists as long as the participants have no reason to modify the external manifestations of emotion because of social circumstances. Hence, it is important to study objective as well as subjective indicators of facial expressions because that may provide an indication of the conscious and unconscious emotional state of individuals.

One of the most important ways in which we express emotions is through facial expressions. Facial expressions are movements of the muscles supplied by the facial nerve that are attached to and move the facial skin. These externally visible indicators of emotions may serve as an objective measure of emotion. Accordingly, facial expressions have been used to study emotions in areas including education, ^{39 40} artificial intelligence research, ^{41 42} psychology, ^{43 44} and many other fields. The study of the relationship between facial expressions and emotions dates back to Darwin's research in 1872 resulting in the later published book, “The expression of emotion in man and animals.” ⁴⁵ They claimed that facial expressions are external manifestations of emotional responses occurring in combination with other bodily responses such as vocalizations, gestures, skeletal movements, and physiological responses. They further described that all people, regardless of their race or cultural origins, express emotions in the same manner. This characteristic of facial expression, called universality, was studied by several researchers since Darwin. ^{33 46 47 48 49} All automatic facial expression recognition algorithms work based on the universality of facial expressions. However, this property of universality has been debated by many researchers showing that facial expressions may have different meanings based on culture and that the intensity of facial expressions may also depend on cultural norms. ^{50 51 52 53} Automatic facial expression recognition algorithms try to reconcile these differences by training their machine learning algorithm with facial expressions from a variety of cultural groups during their training and validation phases as was the case with the algorithm used in the present study.

Besides being an indicator of internal emotional states, facial expressions of emotion can also indicate an individual's behavior (i.e., an angry facial expression may be a precursor to later harmful behavior). ⁵⁴ Moreover, facial expressions are also seen to evoke complementary or other emotional responses from the communication partner (i.e., sad facial expressions may evoke sympathy in the communication partner). ^{55 56} Hence, facial expressions of emotion can influence a communication situation in multiple ways, making it more important to study them, especially in adverse listening situations where individuals with/without hearing loss already face communication breakdowns.

Facial Expressions, Listening Difficulty, and Listening Effort

Individuals with hearing loss face communication breakdowns in adverse listening situations such as understanding speech in noise. Understanding speech in noise is complex and requires multiple levels of peripheral and higher level processing involving activities such as segregation of information from nonessential background signals, maintaining attention on the target message, decoding the message, and committing it to memory. ⁵⁷ A breakdown in any of the levels of processing could potentially cause cognitive disequilibrium. Cognitive disequilibrium is a state of uncertainty caused when situations present obstacles to goals, and when individuals experience contradictions, discrepancies, and/or gaps in knowledge. ⁵⁸ Such situations may evoke emotions of confusion and frustration with the accompanying facial expressions. Francis and Oliver ⁷ also noted that it is more likely to see emotions of frustration and annoyance in individuals with hearing loss to adverse listening situations than stronger emotions like anger or fear. We propose that measuring these emotional responses may provide a measure of the listening difficulty experienced by individuals in challenging listening conditions. In a qualitative interview, Marinelli ⁵⁹ asked individuals with hearing loss about their emotional consequences of applying listening effort (“Are there emotional impacts of your hearing loss/listening effort?”) during communication. Participants reported a mix of fear, frustration, sadness, stress, embarrassment, doubt, and anger in various communication situations as the emotional consequences of their listening effort. They attribute this to the increased effort expended by them as well as the feeling of missing out parts of conversations. This suggests a possible link between emotions and listening effort. Furthermore, Francis and Love ² discussed the extensive overlap between physiological systems associated with listening effort and affective responses and describe listening effort as an affective and cognitive phenomenon. They examine different measures of listening effort and conclude that there is an unresolvable interaction between the commitment of processing resources and the affective responses observed during their deployment. Hence, they remark that it is important to study the affective aspects of listening effort which may later assist in explaining communication-related decisions of participating in a social situation, use/non-use of hearing devices or hearing protection, and may relate to the long-term health of individuals. Listening effort generally increases with increases in listening difficulty. ^{4 10 13 60} Hence, studying facial expressions at different levels of listening difficulty may provide an indication of the emotional responses at these different difficulty levels that may relate to the listening difficulty experienced by individuals. In the current study, we change the listening difficulty by varying the SNR.

Theories of Emotion and Facial Expression

There are two main schools of thought that examine emotions and facial expressions currently. Ekman and Friesen ⁶¹ classified emotions in six families: anger, joy, fear, sadness, surprise, and disgust. He called these emotions “basic” and claimed that the basic features of all these emotions (1) are universal, i.e., they have a universal facial/vocal expression; (2) are present in other animals; (3) have distinctive patterns of autonomic nervous system activity; (4) have universal antecedent events; (5) have coherence in response systems; (6) have quick onset; (7) have brief duration; (8) have unbidden occurrence; and (9) have automatic appraisals. This theory is also called the discrete emotions theory and states that emotional situations give rise to specific affect programs for basic emotions that produce prototypical responses which include emotion-specific facial expressions. A school of thought that challenges the idea of basic emotions is the appraisal perspective. ^{62 63 64} Also called the componential models of emotion, ⁶⁵ these models postulate that facial expressions consist of components which differ based on an appraisal of the situation. In these models, it is proposed that emotions are based on a process that includes cognitive activity, motor expression, physiological arousal, action tendencies, and subjective feeling states. The proponents of these models are not convinced that muscle movement combination alone can characterize emotions and believe that context is necessary to interpret these emotions. ^{62 66} They also believe in the existence of a large variety of emotions, much more than the basic emotions proposed by the discrete theorists.

In the context of studying emotional responses in adverse listening conditions, we are interested in identifying the emotions displayed by individuals during these conditions. Emotions in adverse listening situations have not been studied. However, situations where individuals experience cognitive disequilibrium or gaps in information have been studied in the field of education and psychology. It is seen that emotions such as confusion and frustration were commonly seen during complex problem-solving situations as well as during the process of learning. ^{39 42 67 68} These emotions were seen when individuals experience cognitive disequilibrium, or encounter gaps in knowledge. In response to these challenging stimuli, the autonomic nervous system increases its arousal and learners experience emotions such as confusion, frustration, irritation, and anger.

We know that understanding speech in noise is a complex process which requires multiple levels of peripheral and higher-level processes. ⁵⁷ We hypothesize that speech in noise situations may similarly give rise to emotions of confusion and frustration due to the presence of obstacles to the goal of speech understanding, and presence of gaps in knowledge due to missing words in noise. When individuals encounter these complex situations, confusion and frustration may be aroused with the associated facial expressions. Facial expression recognition algorithms can automatically measure these facial expressions. We hypothesize that with an increase in listening difficulty, the probability of the presence of the presence of these expressions will increase. This may relate to the listening effort employed by the individual and provide an indication of the affect component in the speech perception task. If found to be feasible, this would provide a relatively easy measure of listening difficulty and the affect component in the situation that can be carried out simultaneously with speech perception testing, with minimal additional equipment (camera) and training. The purpose of this study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. Since earlier literature has found that facial expressions of confusion and frustration are seen when individuals encounter gaps in knowledge or when the cognitive equilibrium is disturbed, these were our candidate facial expressions. Our hypothesis was that when difficulty in adverse listening situations increases, listeners would be more likely to exhibit facial expressions that depict confusion and frustration. A secondary aim was to determine if self-perceptions of emotions correlate with results of automatic facial expression detection software.

Method

Participants

Thirty-four participants with normal hearing sensitivity participated in the study. Hearing thresholds were measured at octave frequencies from 250 to 8,000 Hz. Normal hearing was defined as pure tone air conduction thresholds of ≤25 dB HL for octave frequencies in both ears. The data of one of the male subjects were excluded because he was unable to complete the study due to time conflicts. The remaining 33 participants (16 males, 17 females) were between the ages of 18 and 34 years (mean: 23, SD: 4). A sample size of 32 participants was estimated using our pilot data (Venkitakrishnan, Guiliani, & Wu). ⁶⁹ We conducted sample size estimation using the simR package in R at an alpha = 0.05. This sample size resulted in a power of 0.81 (95% CI: 71.93–88.16). All participants were native speakers of English. Racially, 30 subjects identified as white, and three as Asian. Two subjects also reported their ethnicity as Hispanic/Latino. Participants were recruited through mass email from the University of Iowa community. Fourteen of our participants had normal vision, nine had contact lenses, and ten had glasses. The testing was done without glasses on. Contact lenses were left on. Our exclusion criteria included participants with self-reported emotional or psychological disorders, with history of facial paralysis, or history of mood-affecting drug use. All participants completed the Hospital Anxiety and Depression Scale. ⁷⁰ This scale is a self-assessment scale that detects states of depression and anxiety. Though this scale was first developed to assess these mental states in hospital/medical outpatient setup, it has been widely used in general clinical practice as well as in research. ⁷¹ Scores of seven or less in both the anxiety and depression subscales indicate noncases. We excluded participants who scored more than seven in either subscale from participation in the study because we did not want existing emotional states to impact their facial expressions in the study. The included participants had a mean score of 4.4 (range: 3–6, SD: 1.14) for the anxiety subscale and a mean score of 1.2 (range: 0–4, SD: 1.78) for the depression subscale. All procedures were approved by the Institutional Review Board of the University of Iowa. The participants were compensated for their time. This study was part of a larger study evaluating the effect of simulated hearing loss and listening difficulty on emotional responses.

Materials

Speech Material

IEEE sentences ⁷² were presented in quiet and in different SNRs. We used the SNRs of −1, +2, +5 dB SNR relative to an individual participant's SNR-50 level. SNR-50 is the SNR at which the participant could understand 50% of speech and we obtained this using an adaptive SNR procedure (explained later).

Prior to the experiment, we had conducted a pilot study with five participants to determine if the IEEE sentences by themselves caused any facial expressions of emotions. For this, we presented all 72 lists of IEEE sentences in quiet and recorded and analyzed participants' facial expressions using the method described later. The participants quietly listened to all the sentences. While they were listening, their facial expressions were monitored. If any sentence in a list was found to evoke emotions of joy, anger, surprise, guilt, sadness, confusion, or frustration, the whole list was excluded from use in our experiment. We did this to ensure that the emotions seen in our experimental conditions result from experimental manipulations and not from the emotional content in the sentences.

Facial Expressions

Facial expressions were recorded using a camera (Logitech HD Pro Webcam C920) and analyzed using the Emotient FACET software (v8.0; iMotions).

The FACET software is based on the Facial Action Coding System or FACS. ^{73 74} FACS is an established method of classifying facial expressions and assigning emotions to these expressions based on the combination of different facial movements. FACS is an anatomy-based method of describing facial movements based on movements of different muscles of the face. ⁶¹ These muscles could be single muscles or groups of muscles that work together to make a facial movement possible. Each muscle or muscle group that is responsible for a specific movement was labelled as an action unit. FACS in itself is just a measurement system, and it does not interpret the meanings of facial movements. However, researchers can use the action units to detect facial expression and thereby, the emotion conveyed by a person. Nevertheless, this is a very time-consuming process and manual coding by coders trained in FACS takes around 24 minutes for 1 minute of video data. ⁷⁵ Automatic facial expression analysis systems considerably reduce this time and effort.

The FACET software used in this study automatically detected and identified facial expressions based on its proprietary algorithm. This software first detected the face and then identified different landmarks on the face such as eyes, eye corners, lip corners, mouth, and mouth corners. Following detection of the face and facial movements, the algorithm compared the movements of the participant's face during our experiment with its normative database. As the participant's face moved and created a facial expression, the movements were interpreted based on the FACS. Depending on the synergistic movement of the different action units, the algorithm then computed the evidence level , which is the probability of the presence of a given facial expression. The software also combined different action units and provided an evidence level for different emotions. FACET software provided the evidence level in log-odds unit. Hence, for confusion, an evidence level value of 1 means that it is ten times more likely that the facial expression seen in the subject would be classified as confused than not confused by an expert human coder. Similarly, value of −2 would mean that it is 100 times more likely that the value be classified as not confused than confused by an expert human coder. The general range of the evidence values is from +4 to −4. In the current study, we analyzed and present the emotion of confusion and frustration because these emotions are likely to be present in an adverse listening situation.

The algorithm developed by iMotions to detect facial expressions was based on different datasets which contained facial expressions of individuals obtained through posed and spontaneous emotions in different environments, using different positioning of the face, and using individuals of different ages and ethnicities. These datasets are (1) extended Cohn-Kanade dataset, ⁷⁶ which contains 593 videos from 123 individuals aged 18 to 50 years from the Euro-American, Afro-American, and other groups. Their dataset consisted of posed and spontaneous emotion videos. (2) LEAD database was internally developed by iMotions and contains webcam recordings in different home environments and lighting conditions. (3) Multi-PIE database ⁷⁷ contains data from 337 individuals recorded in 15 viewpoints and under 19 illumination conditions in four different recording sessions. iMotions reports an accuracy of 80 to 100% in identifying emotions using these three datasets (white paper from iMotions). Independent studies also found that Emotient FACET has an accuracy of 97% for identifying pictures of emotions, which is better than average human coder performance (60–80%) and around 65% for spontaneous emotions. ⁶⁴ A strong correspondence was found between FACET-predicted emotions and human coders' predictions of emotions. ^{78 79} These studies used both spontaneous and posed facial expressions obtained from research participants, as well as well-established facial expression databases. Another study also showed significant correlations between human and FACET-predicted basic emotion identification with large effect sizes for both facial expressions (large effect side: Cohen's d  > 1.71) and for action units (medium to large effects: > 0.72). ⁸⁰ Different studies have found high action unit detection accuracy in addition to good facial expression recognition accuracy, which is a good indicator for recognition of other facial expressions that are based on the same action unit (e.g., facial expressions of confusion detected with the action unit 4 [eyebrow lowerer] movement). This same action unit is also involved in emotions such as confusion, frustration, and anger.

Subjective Listening Effort Rating Scale

We also asked the participants to rate their subjective listening effort using the first question in a 10-point scale adapted from Picou and Ricketts. ⁶⁰ We asked the participants to rate their response to the question “How hard did you work to understand what was said? Remember, this is different from how many words you got right. For example, you could get all the words right, but have to work very hard to do it.” Participants rated their listening effort after every block of five sentences. A score of 0 meant they had to work “very hard” and a score of 10 meant they worked “not at all hard.”

Self-Rating of Emotion

We used the Epistemically Related Emotion Scale ⁸¹ for self-rating of emotions. Emotions that are triggered by cognitive characteristics of tasks such as surprise, confusion, and curiosity are called epistemic emotions. We asked participants to rate their subjective feelings of confusion and frustration, using a scale of 1 (not at all) to 5 (very strong) after every block of five sentences. We used these subjective ratings for comparison with the objective emotional responses from the automatic facial expression recognition software.

Procedure

First, we obtained an informed consent from the participant. We then determined the participant's eligibility for inclusion in the study by asking them to fill out the Hospital Anxiety and Depression Scale and obtaining their hearing thresholds. If the participant was eligible to continue, we obtained their individualized SNR-50 using IEEE sentences as the stimuli. An adaptive procedure similar to the Hearing in Noise Test was used to calculate the individualized SNR-50. ⁸² Here, the participants were asked to listen to IEEE sentences presented at 65-dB SPL with speech-shaped noise. The speech level was adjusted adaptively using the one-down, one-up procedure in 2-dB steps. We considered a response to be correct if all words were repeated accurately, with exceptions for a/the and is/was. Twenty IEEE sentences were played, and the last 17 sentences were used for determining the individual SNR-50 levels. This average −2 dB was defined as an individual's SNR-50. This was done so that at 0 dB SNR, the listener's performance would be close to 50% if the scoring in the experiment was based on words. ⁸

The SNRs used in this study (i.e., −1, +2, +5 dB SNR) were relative to this individualized SNR-50 for each participant. We presented sentences at a level of 65 dB SPL in quiet and in IEEE-shaped noise. We chose these SNRs so that the conditions would produce different difficulty levels resulting in scores from 50 to 100% ⁸ where measurement of listening effort is more critical as speech intelligibility is present. We also wanted to assess listening effort in more realistic SNRs and wanted to avoid participants giving up due to increased difficulty of the condition. We considered our changing SNRs as varying levels of listening difficulty from the easiest condition, quiet, to the most difficult condition in our experiment, −1 dB SNR. We used a stationary noise masker with the long-term average speech spectrum of the IEEE sentences. We kept the noise level constant and varied the speech level to change the SNR to ensure that noise did not become too loud at low SNRs, and so that the noise level did not cue participants to predict task difficulty when the noise began. Twenty sentences were presented using a custom MATLAB code in each of the SNRs and in quiet. They were presented in random blocks of five sentences of different SNR. Each sentence was followed by a 2-second retention period. A brief tone was presented following this retention period to prompt the participant to repeat the sentence. They were given points for each word repeated correctly and an average of the percent correct scores was used for statistical analyses. After each block of five sentences, participants subjectively rated their listening effort and their emotional response with the rating scale described earlier over those sentences. In this way, subjective rating for the same SNR was obtained four times (four blocks of five sentences each = 20 sentences) and an average of these was used for analyses. We recorded the facial expressions for the entire duration of the testing using our webcam linked with the iMotions software. The time course of the first trial of each condition is shown in Fig. 1 .

Figure 1.

Time course of the first trial of a speech in noise condition. Here the noise starts at 0 seconds. The first second of noise (or quiet in quite condition) is considered the baseline for the facial expression analyses. This is followed by 4 seconds of sentences that are offset-aligned. Then participants wait for 2 seconds (retention period). At the end of these 2 seconds, a prompt is given after which participants repeat the sentence.

We conducted the experiment in a sound attenuating booth. The stimuli were presented using ER-3A insert earphones in both ears in a diotic listening setting. The auditory stimuli were generated by a computer and Motu UltraLite audio interface. This was routed to a GSI-61 audiometer and presented using the ER-3A insert earphones. The output of the insert earphones was calibrated in a 2-cc HA1 coupler using a Larsen Davis System 824 sound level meter. This experiment was a part of a larger project to measure the effect of hearing loss on facial expressions and pupil response. Participants were seated at approximately 60 cm away from the camera. They were asked to rest their head on a chin rest. This was done so that we could also record their pupil responses simultaneously (used for the larger study). MATLAB was used to present stimuli and time stamps to the iMotions software. Time stamps were presented at the start of noise, start of sentence, end of sentence, and when the response prompt was presented.

Data Processing

Facial expression analysis : First, we applied baseline correction to the evidence values of each sentence. Each participant's baseline confusion value was an average of the instantaneous confusion values seen for the 1 second of noise presentation before the start of the sentence. For the quite condition, this baseline is collected in quiet. This is done because we wanted to calculate confusion values to a combination of speech and noise and wanted to control for any emotion due to the presence of noise alone. Following baseline correction, we obtained the mean confusion values for the sentence + noise and the retention period for each sentence for each participant. We removed the values that were greater or less than 3 standard deviations of this mean value for a given participant. We interpolated the missing values. We then plotted the time course of this response. We used the positive area under the curve (AUC) for confusion during the retention period (4–6 seconds) for further analyses. We used the positive AUC because this area indicates an increased probability of the presence of the emotion. ⁷⁴ We analyzed the data within the retention interval because previous studies in pupil response analysis have shown that this interval shows effect of memory load as well as speech perception causing maximal listening effort to be seen during this interval. ^{83 84} AUC was obtained for confusion and frustration for the different conditions. This was averaged across participants. These AUC values were used as the emotional response and for statistical analysis.

Statistical Analysis

To determine if speech recognition scores, objective emotional responses, subjective rating of emotion, and subjective listening difficulty increase as the listening difficulty increases, we used separate linear mixed effects analysis with emotional response as the dependent variable. Our fixed effect was SNR, and we added a random intercept for subject. We conducted pairwise post hoc comparisons using Tukey test with correction for multiple comparisons.

To investigate the relationship between subjective and objective measures, we used the correlations package ⁸⁵ and used multilevel correlations. Multilevel correlation here is a special case of partial correlations that accounts for repeated measures by considering SNR as a random effect in a mixed linear regression. Our predictor variables were subjective measures of emotion and the listening effort rating, and dependent variable was objective emotional responses of confusion.

Results

The purpose of our study was to determine if facial expressions can be used as an index of listening difficulty and to determine the relationship between subjective and objective measures of emotional responses and listening effort.

Speech Recognition Scores

Fig. 2 shows the percent correct scores of for quiet and for different SNRs. Though these scores are represented in percentage, the statistical analysis was done using the rationalized arcsine transform scores to stabilize error variances near the extreme. ⁸⁶ We saw a significant main effect of SNR (F _(3,96)  = 451.16, p  < 0.001). Furthermore, pairwise post hoc comparisons revealed that all the pairwise comparisons were significant ( p  < 0.001). This shows that there was a significant decrease in percent correct scores as the difficulty level increased from the quiet to +5 dB SNR to +2 dB SNR and to −1 dB SNR.

Figure 2.

Average speech recognition scores from the most (−1 dB signal to noise ratio [SNR]) to least difficulty (Quiet) level. The main effect of SNR is significant and all pairwise comparisons were significant. Levels of significance: *< 0.05; ** < 0.01; *** < 0.001.

Facial Expressions

We used the emotions of confusion and frustration and measured these facial expressions. Fig. 3 shows the average evidence of confusion with the progression of the sentence or the time course of the emotional response. Here, zero seconds denotes the start of the sentence. Each sentence was off set aligned at four seconds which means that all sentences ended at four seconds. Four to six seconds denotes the rehearsal/retention time, and a response prompt is presented at 6 seconds after which the participant repeats each sentence. The y -axis denotes the evidence of confusion represented in the log-odds. Recall that a value of 1 would mean that it is 10 times more likely that the facial expression is identified as confused than not confused by an expert human coder. Similar trend is also seen for the emotion of frustration. We then calculated the AUC for each sentence. Fig. 4 shows the AUC of the confusion response during the rehearsal period (4–6 seconds) for quiet and for each SNR for all participants. When the participant repeats the sentence, it is not possible to monitor facial expressions because the lip movements interfere with the facial expression recognition. We saw a significant main effect of SNR (F _(3,96)  = 7.89, p  < 0.001) on the AUC of confusion. Further pairwise comparisons showed that the AUC of confusion was significantly greater for the −1 dB SNR compared to the +5 dB SNR ( t  = 3.216, p  = 0.0094) and the quiet condition ( t  = 4.670, p  < 0.001). The AUC of confusion was also significantly greater for the +2 dB SNR as compared to quiet condition ( t  = 2.788, p  = 0.0319). The rest of the comparisons were not significant. Similar trends were seen for the emotion of frustration. We saw that the main effect of SNR was significant (F _(3,224)  = 11.743, p  < 0.001). We also found that the AUC for the −1 dB SNR was greater than the +5 dB SNR ( t  = 4.157, p  < 0.001) and the quiet ( t  = 5.645, p  < 0.001) conditions, and the AUC for frustration was greater for the +2 dB SNR condition compared to the quiet condition ( t  = 3.187, p  = 0.0089). This shows that confusion and frustration displayed by participants increased; however, it was not significantly different with change in each difficulty level.

Figure 3.

Time course of the evidence of confusion (averaged across sentences and participants) across the different signal to noise ratios and in quiet. Sentence and noise are present from 0 to 4 seconds. 4 to 6 seconds is the retention period.

Figure 4.

Area under the curve (AUC) for confusion (averaged across sentences and participants) for quiet and various signal to noise ratios (SNRs). The main effect of SNR is significant and the pairwise comparisons indicated in the figure were significant. Levels of significance: *< 0.05; ** < 0.01; *** < 0.001.

Due to the similar trends seen for the emotions of confusion and frustration, we also determined the correlation between confusion and frustration emotions. Fig. 5 shows the relationship between the AUC for confusion and frustration. We see that both these parameters have a strong positive correlation ( r  = 0.97, p  < 0.001). Thus, we focus on the emotion of confusion. However, similar trends were seen with the emotion of frustration.

Figure 5.

Correlation between the evidence of confusion and frustration (objective measures of emotional response represented using their area under the curve or AU). We find that they are positively related. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (Square), +5 dB (circle), and quiet (triangle).

Listening Effort Rating

We also asked participants to rate how hard they needed to work to understand what was said with a rating of 0 for “very hard” to a rating of 10 representing “not at all hard.” Fig. 6 shows the listening effort rating for quiet and the different SNRs. We saw a significant main effect of SNR (F _(3,96)  = 228.57, p  < 0.001) and all pairwise comparisons were also significant at p  < 0.001. Thus, with an increase in difficulty level, participants reported a significant increase in the exerted listening effort.

Figure 6.

Listening effort rating for the different signal to noise ratios (SNRs) and quiet condition. The rating scale goes from 0 representing greatest listening effort to 10 representing least listening effort. These scores are an average across blocks of five sentences with four ratings in each condition from each participant. The main effect of SNR is significant and so are all the pairwise comparisons. Levels of significance: *< 0.05, ** < 0.01, *** < 0.001.

Subjective Emotional Response Rating

We asked participants to rate their subjective emotional responses after every five sentences using a scale of 1 (not at all) to 5 (very strong). The rating for confusion is represented in Fig. 7 . We saw a significant main effect of SNR (F _(3,96)  = 126.35, p  < 0.001) on the subjective rating of confusion. All pairwise comparisons were also found to be significant at p  < 0.001. We also asked participants to rate their frustration after every block of five sentences. Frustration rating results were similar to confusion and we found the effect of SNR to be significant (F _(3,224)  = 107.32, p  < 0.001). Follow-up testing showed all pairwise comparisons between the different SNRs were significant. This showed that participants rated feeling more confused and frustrated as the difficulty level increased.

Figure 7.

Subjective confusion rating for quiet and the different signal to noise ratios. A rating of 1 represents not at all confused to 5 representing very strong confusion. These scores are an average across blocks of five sentences with four ratings in each condition from each participant. The main effect of condition and all pairwise comparison are significant. Levels of significance: * < 0.05, ** < 0.01, *** < 0.001.

Correlation of Objective and Subjective Measures

Fig. 8 shows the AUC for confusion in relation to the self-rating of confusion. We see that these parameters have a weak positive correlation ( r  = 0.20, p  = 0.024). As the confusion rating increases, the AUC of confusion also shows an increasing trend. Fig. 9 shows the AUC for confusion in relation to the listening effort rating. Again, we see that the listening effort rating significantly predicts the confusion AUC ( r  = −0.27, p  = 0.002). As the listening effort rating is higher (representing less effort), the AUC of confusion decreases.

Figure 8.

Relationship of subjective self-rating of confusion and objective area under the curve for confusion. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (square), +5 dB (circle), and quiet (triangle).

Figure 9.

Relationship of listening effort rating and objective area under the curve for confusion. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (square), +5 dB (circle), and quiet (triangle).

Discussion

The purpose of the study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We monitored facial expressions of participants to speech in quiet and to speech in noise. We also asked participants to report their perceived listening effort and emotional responses.

Facial Expressions as a Measure of Listening Difficulty and Emotional Responses

We found that the emotions of confusion and frustration increased as the listening difficulty increased. Confusion and frustration emotions are characterized by the movement of the action unit 4, which is the brow lowerer. ⁴² Muscles responsible for movement of the eyebrows include the frontalis muscle with some contribution from the corrugator muscle. Mackersie and Cones ¹⁶ studied the frontalis muscle during a dichotic digit task and found a significant relationship between task difficulty and frontalis muscle activity. They saw a significant increase in electromyographic activity from their medium- to high-demand tasks, but not for low- and medium-demand tasks. Similarly, in our study, we found a relationship between task difficulty and emotional response, and significant differences between alternate difficulty levels. The advantage of our approach was its noninvasiveness compared to the use of surface electrodes for facial electromyography which may also affect facial expressions due to its intrusiveness.

Our results also show that speech recognition scores decrease with decreasing SNR. Accordingly, perceived listening effort also significantly increases with increasing difficulty level. Facial expressions of confusion and frustration show an increase with increasing listening difficulty; however, it may not be as sensitive as the other measures in the study. We see that the evidence of confusion/frustration increases when moving from quiet to +2 dB SNR, quiet to −1 dB SNR, and from +5 to −1 dB SNR. However, there is no significant difference between adjacent difficulty levels. This could be because a larger change in difficulty level is needed before facial expressions change significantly. This is contrary to measuring listening effort using pupil response measures where pupil changes are seen consistently with changes in speech intelligibility, ^{4 21} and often without changes in speech intelligibility. ^{17 87} The sensitivity of facial expressions could be compared to skin conductance or heart rate variability measures. Skin conductance has shown varying sensitivity to listening effort, showing no change with changing SNR, ^{12 14} but changes to more extreme changes in task difficulty ¹⁵ and changes when changing from quiet to speech in noise conditions. ¹⁴ Heart rate variability also showed sensitivity to more difficult tasks such as speech recognition performance levels of lower than 60%, ¹⁴ complex tasks such as dichotic digit recall and more difficult SNRs of +2 or +5 dB, ¹⁵ and for increased task demands required during fast rate of speech. ⁸⁸ Some drawbacks of the skin conductance and heart rate measures is that they can be influenced by general stress. Heart rate measures also require invasive electrode placement and are also influenced by changes in breathing such as those required for verbal responses during a sentence repetition task. Facial expressions are noninvasive and can be used to assess the time course of emotional responses related to listening difficulty.

Francis and Oliver ⁷ provided a model showing interactions between effort, motivation, and affective systems and emphasized that affect systems play a role during speech perception in challenging listening situations. They also argue that the effort and motivation applied in a communication situation is mediated by the individual's emotional state. ^{89 90} From the FUEL model, we know that effort expended in daily life communication situations depends on the listener's motivation. ¹ Other studies have also shown that motivation affects individuals' listening effort and their participation in communication. ^{60 91 92} Individuals with hearing loss also weigh the listening effort cost to the social connectedness benefit to determine whether to participate in a communication situation and may be more motivated to participate if the cost of the effort is worth the social connectedness benefit. ⁶ Furthermore, in a study where participants were provided monetary reward in lieu of increased effort, the researchers noted that the role of positive emotions on pupil response could not be discounted. ⁹¹ Alternatively, it is also seen that the presence of another individual during listening tasks causes an increase in listening effort. ^{93 94} The researchers state that increased motivation due to social evaluative stress to perform could be one of the reasons for this increased effort. Hence, emotions, motivation, and affect all have an impact on listening effort and may impact an individual's engagement and participation in communication. Thus, it may be important to understand an individual's affective state to try to predict their engagement in communication. Our results from the objective and subjective measures of emotional responses show that emotional responses do occur during the process of an adverse listening situation and increase as the listening difficulty increases. This emotion may play a role on individual's willingness to participate in a communication situation. Moreover, their recollection of the emotions perceived in a communication situation may impact their future participation in what they feel is an adverse listening situation. Emotionally charged situations are known to be more memorable and research also shows that this occurs more for situations inducing negative affect. ^{95 96 97} Emotional responses in difficult listening situations may have varying implications on whether to continue wearing hearing aids, or on engagement in a social situation and these decisions could impact an individual's long-term mental health. ⁷ Past research shows that emotional response to non-speech sounds were related to feelings of isolation and social disconnectedness. ²⁷ Individuals with hearing loss already experience negative feelings of distress, depression, and loneliness. ⁹⁸ If they experience more negative emotions as a result of the challenging listening situation, this may cause them to avoid those communication situations. ¹¹ Long-term consequences of effortful listening may depend on the emotional responses of individuals. Hence, it is important to study these responses and determine if hearing rehabilitation is reducing these negative feelings. Future studies should explore the extent of the different emotions present in individuals with hearing loss in adverse listening situations and study the impact of these emotions on their quality of life.

Objective and Subjective Measures of Emotional Response

We saw that both the subjective emotional response rating and the listening effort rating were able to significantly predict the objective AUC of confusion. Thus, with increasing listening difficulty or subjective listening effort, we can see corresponding changes in objective emotional response. This is contrary to what is seen in the listening effort literature where no correlation between objective and subjective measures of listening effort is seen generally. ^{14 15 16 21} Cognitive and affective constructs may influence the same physiological systems. ⁷ This could mean that listening effort and the affective system are interrelated. This is seen in the relationship between listening effort rating and the confusion AUC ( Fig. 8 ). We do not see a strong association between subjective and objective emotional response measures. One reason for this could be that individuals often do not like to display negative facial expressions perhaps due to the negative connotations associated with these facial expressions. ⁹⁹ Another reason could be because the intensity of confusion displayed by individuals may be different. Though individuals experience confusion/frustration, their display in the form of facial expressions may be influenced by display rules of the different cultures they belong to, ^{100 102 101} or due to their life experiences. Individuals who are exposed to cultures where emotions are internalized and not externally displayed may show subtle changes in facial expressions even though they rate having experienced them. It is possible to reduce some of this variability by normalizing or standardizing emotional responses. This can be done by determining the lowest and highest evidence value of confusion/frustration experienced by each participant and then converting this to a predetermined scale (e.g., minimum evidence is transformed to 0, and maximum to 100), and then comparing the change in emotion seen in each individual. Furthermore, emotions may be present even in the absence of identifiable facial expressions because the facial expression of confusion is a subtle emotional response. This could explain why the subjective rating changes significantly with each difficulty level than the objective facial expressions. Emotional responses in the form of facial expressions supplement information about the unconscious affect component in addition to the felt/conscious affect component provided by the subjective rating scales. Hence, it is worth measuring both these constructs of emotional response. Facial expressions can also be recorded simultaneously with pupil responses. Hence, despite the decreased sensitivity, this method holds promise for further investigation as a measure of objective emotional responses and listening difficulty. If measured along with pupil responses, this may provide us both with a more sensitive measure of listening effort and evidence of emotional responses in response to different listening situations.

Clinical Implications

Measuring facial expressions could help provide an objective/unconscious measure of emotional response and listening difficulty experienced by an individual in different adverse listening situations. Like other measures of listening effort, facial expressions can also be measured, for example, with different hearing aids or different hearing aid features to aid in choosing a hearing aid/hearing aid feature. The task involved is relatively easy and does not require complex directions. Since facial expressions during speech are not used, this can potentially be investigated as a measure not requiring response from participants and may have utility in difficult-to-test populations such as individuals with dementia or other cognitive or speech/language deficits precluding them from responding verbally to make decisions about different hearing aids/hearing aid settings. This would require work to determine if facial expressions in these populations are similar to the automated facial expressions seen in individuals without cognitive impairments. This method may also have applications in monitoring confusion/frustration or other negative emotions in children with hearing loss. Emotions monitored during the process of learning may aid in changes to hearing aid settings, instructional strategy, or the environment to improve auditory access until the negative emotions have been reduced. Finally, measuring listening effort using facial expressions is a natural and easy measure that can be administered simultaneously with speech perception testing with minimal additional equipment (camera) and training. Though the software used in the present study was licensed, there are open-source facial expression algorithms available that can be used free of charge.

Limitations

One of our hypotheses is that individuals will display the emotions of confusion and frustration when faced with difficulty with understanding speech in noise. In real-world scenarios, it is possible that they may try to mask this confusion by other compensatory strategies such as nodding or expressing understanding despite failure to understand. Since the experimental scenario did not have similar social pressures or cost for not understanding speech, we assume that such strategies would be unnecessary. However, these considerations must be made if this measure is used in the real world. The necessity of larger increments in SNR needed before changes in facial expressions are seen could mean that facial expressions are not as sensitive to changes in listening difficulty as other measures such as pupil responses or dual-task paradigms. Furthermore, our data were collected in individuals who predominantly identified as white. Though the facial expression algorithm we used was trained on a diverse population, further study is required to ensure that these results generalize to diverse racial and ethnic groups.

Conclusion

The purpose of our study was to determine if facial expressions can be used as an index of listening difficulty and emotional responses. We see that the facial expressions of confusion and frustration increase with increases in task difficulty. This shows that this can be further explored as a measure of listening difficulty and affect. Confusion and frustration emotions showed very similar trends which may mean they measure the same aspect of the objective emotional response. Further, there is a significant relationship between the objective measure of confusion and the self-rating of confusion, and self-rating of listening effort. This shows that affect measures and measures of listening effort have similar effects on individuals with regard to effects on speech understanding in noise.