Representation of expression and identity by ventral prefrontal neurons

Abstract

Evidence has suggested that the ventrolateral prefrontal cortex (VLPFC) processes social stimuli, including faces and vocalizations, which are essential for communication. Features embedded within audiovisual stimuli, including emotional expression and caller identity, provide abundant information about an individual’s intention, emotional state, motivation, and social status, which are important to encode in a social exchange. However, it is unknown to what extent the VLPFC encodes such features. To investigate the role of VLPFC during social communication, we recorded single-unit activity while rhesus macaques (Macaca mulatta) performed a nonmatch-to-sample task using species-specific face-vocalization stimuli that differed in emotional expression or caller identity. 75% of recorded cells were task-related and of these > 70% were responsive during the nonmatch period. A larger proportion of nonmatch cells encoded the stimulus rather than the context of the trial type. A subset of responsive neurons were most commonly modulated by the identity of the nonmatch stimulus and less by the emotional expression, or both features within the face-vocalization stimuli presented during the nonmatch period. Neurons encoding identity were found in VLPFC across a broader region than expression related cells which were confined to only the anterolateral portion of the recording chamber in VLPFC. These findings suggest that, within a working memory paradigm, VLPFC processes features of face and vocal stimuli, such as emotional expression and identity, in addition to task and contextual information. Thus, stimulus and contextual information may be integrated by VLPFC during social communication.

Communication is the substrate of our social interactions, with faces and vocalizations being the primary elements of social communication. Faces carry a wealth of information including the identity of the speaker, age, gender, emotional state and situational or contextual factors (Knappmeyer et al., 2003; Bruce and Young 1986; Haxby et al., 2002; Jack and Schyns, 2015). When facial information is combined with accompanying speech or vocalizations, the information is strengthened and clarified (Yovel and Belin, 2013; Campanella and Belin, 2007; Ghazanfar and Logothetis, 2003). The social communication network in the human brain includes regions of the temporal lobe for processing face and voice information (Kanwisher et al., 1997; Belin et al., 2000; von Kriegstein et al., 2005; 2008; von Kriegstein and Giraud, 2006). A similar network has been hypothesized to exist in nonhuman primates and to include cortical and subcortical regions involved in perceptual and cognitive social processing sites in the temporal and prefrontal cortex, as well as the amygdala and periaqueductal gray (Shepherd and Freiwald, 2018). Neurophysiology and functional imaging studies have described specialized regions, i.e. patches, in the temporal lobe of monkeys that are activated by faces (Perrett et al.,1982; Desimone et al., 1984; Tsao et al., 2008; Meyers et al., 2015; Ku et al., 2011; Shepherd and Freiwald, 2018; Livingstone et al., 2017; Dubois et al., 2015; Gothard et al., 2007; Eifuku et al., 2011). In addition, “voice-sensitive” regions, which were first demonstrated with fMRI in the human temporal and frontal cortex (Belin et al., 2000; Fecteau et al., 2005; Pernet et al., 2015), have also been shown to exist in nonhuman primates. In macaques, a “voice-sensitive” region was localized to the anterior temporal lobe and has been shown to be more responsive to vocal than non-vocal stimuli. (Petkov et al., 2008; Perrodin et al., 2011; Joly et al., 2012; Ortiz-Rios et al., 2015).

A key component of the social communication network, the ventral prefrontal cortex (VLPFC), is responsive to faces, vocalizations and their combination in nonhuman primates. Single-unit recordings in macaque monkeys revealed three locations where face-selective neurons were clustered including area 12 ventrolateral, area 45 and area 12 orbital (O’Scalaidhe et al., 1997; 1999). Functional magnetic resonance imaging (fMRI) studies later confirmed these locations with the discovery of face patches in the same three prefrontal regions (Tsao et al., 2008). Following the discovery of prefrontal face-cells, single-unit recording studies revealed an auditory responsive region of the VLPFC with cells that are responsive to species-specific macaque vocalizations (Romanski and Goldman-Rakic, 2002; Romanski et al., 2005). Neurons in these same regions of VLPFC have been further investigated and were found to be multisensory and to integrate species-specific face and vocal information (Sugihara et al., 2006; Romanski and Hwang, 2012; Diehl and Romanski, 2014).

These face- and vocalization-responsive regions in the temporal and prefrontal cortex are presumably organized into circuits that process and integrate particular features of communication stimuli during social communication (Shephard and Freiwald, 2018; Moeller et al, 2008, Meyers et al., 2015). It has been theorized that distinct parts of the circuit process invariant aspects of faces, such as identity, while other regions preferentially process “changeable” aspects of faces such as gaze, mouth movements and expressions (Haxby et al., 2000; Zhang et al., 2020; Pitcher and Ungerleider, 2021). In the face patch system of nonhuman primates, temporal lobe regions activated by identity selectivity are increasingly found as one moves from posterior to anterior in the inferior temporal cortex, whereas response preference for natural motion, including facial expression, is more commonly found in the dorsal face patches medial dorsal (MD) and anterior fundus (AF) (Moeller et al., 2008; Fisher and Freiwald, 2015). In contrast to the segregation of identity and expression in the cortical face patches, amygdala neurons do not appear to segregate based on expression and identity. Amygdala neurons that were tested with face stimuli that differed by identity and expression often responded to both features (Gothard et al., 2007). At the population level, identity and expression were overlapping.

The degree and manner to which identity and expression are processed in other parts of the social communication circuit, such as VLPFC, is unknown. In humans, PET neuroimaging has demonstrated right inferior frontal lobe activation during the processing of emotional expressions (Nakamura et al., 1999). In addition, patients with damage to the ventral frontal lobe have been shown to have difficulty with face and voice expression recognition (Hornak et al., 1996) and are impaired in discriminating specific emotional facial expressions (Tsuchida and Fellows, 2012). VLPFC is also thought to play a role in emotional regulation as demonstrated with transcranial magnetic stimulation (Zhao et al., 2021; He et al., 2020). In particular, the right VLPFC may be important in social interactions requiring emotional regulation (He et al., 2018).

The convergence of sensory afferents in VLPFC from vocal and face processing regions would allow for the computation of identity or expression. Furthermore, since neurons in VLPFC are multisensory, they may exhibit enhanced processing of identity and expression compared to the unimodal face processing areas of the temporo-occipital cortex. However, these features have not been examined in VLPFC neurons.

The present study was designed to determine whether single-unit activity in prefrontal neurons represented identity and/or expression information. Since prefrontal neurons are robustly active during cognitive tasks of working memory and decision-making (Goldman-Rakic, 1996; Brody et al., 2003; Meyer, et al., 2011), we recorded single-unit activity in VLPFC while rhesus macaques performed an audiovisual nonmatch-to-sample (NMTS) task using static faces combined with the corresponding vocalization as the audiovisual memoranda. We examined neural responses that were task-related or stimulus-related and assessed whether they were induced by changes in identity, changes in emotional expression, or both. Our results demonstrate that VLPFC neurons display both task-related and stimulus-related activity during discrimination of species-specific audiovisual face-vocalization stimuli. They also reveal that identity and expression-responsive neurons were localized to the same region of the anterolateral VLPFC where auditory, face and multisensory responsive neurons have been previously identified. These findings provide new evidence supporting the idea that the VLPFC, which integrates stimulus and contextual information, is part of a brain network involved in social communication.

Methods

Subjects and Apparatus

Extracellular recordings were performed in two rhesus monkeys (Macaca mulatta): one male (10.5 kg) and one female (5.0 kg). All procedures were performed in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and were approved by the University of Rochester Care and Use of Animals in Research committee. Prior to recordings, a titanium head-post was surgically implanted. After training on the working memory task had been completed, a titanium recording cylinder was placed over the VLPFC using MRI guided coordinates as previously described (Hwang and Romanski, 2015; Diehl and Romanski, 2014).

Training and recordings were performed in a sound-attenuated room lined with Sonex (Acoustical Solutions, Richmond, VA). During training, the subject sat in a primate chair with the head fixed. Visual stimuli were presented on a computer monitor (NEC MultiSync LCD1830, 1280 × 1024, 60 Hz), which was at 70-cm distance from the eyes. Auditory stimuli were presented via two speakers (Audix PH5-vs speakers; frequency response ±3 dB, 75–20,000 Hz) placed on either side of the monitor at the height of the subject’s head (72 cm distance from speakers to the monkey’s ears). Eye position was continuously monitored with an infrared pupil monitoring system (ISCAN). Behavioral data (eye position and button press) were collected on a PC via PCI interface boards (NI PCI-6220 and NI PCI-6509; National Instruments). The pushbutton was attached to the front of the primate chair just above the hand rest the subjects placed their hand on during the task, providing a short reach to the button. The timing of stimulus presentation and reward delivery was controlled with in-house C++ software, which was built based on Microsoft DirectX technologies.

Auditory and Visual Stimuli

Face and vocal stimuli were taken from short video clips of male and female Macaca mulatta from the University of Rochester created by L.M.R. Face stimuli were extracted from the recorded videos and edited using Adobe Premiere (Adobe Systems, San Joe, CA), Jasc Animation Studio (Corel, Minneapolis, MN), and several custom and shareware programs. The static face images taken from the movie clips were selected to best portray the prototypical facial expression of that particular vocalization expression as described previously (Parr and Heintz, 2009; Hauser et al., 1993; Partan, 2002; Gouzoules, et al., 1984). These were edited (i.e., cropped) to remove extraneous and distracting elements in the background. Our use of static faces, rather than dynamic movies, in this paradigm paralleled the use of static faces in many of the face-processing investigations in the temporal lobe face-patch system. While we have used dynamic movies previously, in the present study, we used static faces obtained from a vocalization movie to control the facial features that were seen by the subjects we recorded from. This also eliminated motion and other extraneous elements that may be present in the full vocalization movie stimulus. Furthermore, it allowed us to direct the subject’s full attention during the stimulus period to the features of this single frame. The objective was to emphasize certain prototypical expressions made during particular vocalizations. Although we carefully selected the facial expression static images from videos of a given identity monkey to reflect only a change in expression, we cannot rule out the possibility that subtle features may differ in the images besides those we elected to study.

While face avatars can provide excellent stimulus control over certain variables and have proved useful in some studies (Chandrasekaran et al., 2011; Sigala et al., 2011; Murphy and Leopold, 2019; Khandhadia et al., 2021), there are many facial and head movements (mouth, eyes, eyebrows, ears, forehead, cheeks, head tilt, shoulders) that occur during different macaque vocalizations/expressions that have not yet been smoothly incorporated into avatar models. Therefore, we used natural stimuli recorded in our home colony, rather than face avatars, in order to preserve the unique features of different facial expressions that may differ across individual macaques.

The corresponding vocalization from the movie was paired with the facial expression to enhance the stimulus and clarify the expression. The audio track of the vocalization was filtered to eliminate background noise, if present, and truncated to remove leading and trailing silence, using Goldwave, MATLAB (MathWorks, Natick, MA) and SIGNAL (Engineering Design, Cambridge, MA). The onset of the vocalization was presented so that it occurred within 5 ms of the static visual image of the facial expression. Sound pressure level of auditory stimuli was adjusted to 65–75 dB (35–112 mPa) at the level of the subject’s ear. All visual images subtended 7–10° of the central visual field and were presented in the center of the computer monitor.

In the NMTS task there are 4 face-vocalization stimuli that are presented as the sample, match and Nonmatch stimuli. The stimuli consisted of two categories of vocalizations/expressions in the macaque repertoire from two unfamiliar monkeys. The affiliative stimuli were coo vocalizations with the corresponding coo facial expression. The aggressive or agonistic vocalizations were barks, threats or submissive screams and their corresponding facial expressions. The vocalization stimuli ranged in duration from 200 to 816 ms. The face and the vocalization stimuli were presented simultaneously. For each block of ~ 200 trials, the 4 face and vocalization stimuli that make up a single set were mixed and matched as Sample and Nonmatch stimuli (Table 1). Each set consisted of an exemplar of an affiliative call (coo) and an exemplar of an aggressive or agonistic (AGG = bark, growl or scream) call, made by a male monkey (M1) and a female monkey (M2) for a total of 4 face/vocalization stimuli (Table 1). These 4 stimuli were mixed and matched as a Sample and Nonmatch in a trial with half of the trials being a change in expression from the Sample to Nonmatch and other half of the trials being a change in identity from Sample to Nonmatch. A single set of 4 face-vocalization stimuli for one block of trials is depicted in Table 1 and Figure 1. We created 4 stimulus sets with 4 face-vocalization stimuli each to use in the audiovisual NMTS task and rotated the 4 sets of stimuli in our testing procedure to reduce the repetitive nature of our testing regimen and to increase attention to the stimuli. Three of the four stimulus sets were used in the recordings.

Table 1:

Trial Conditions and Stimulus Presentations

Condition #	Trial context	SAMPLE	NONMATCH (ID_EXP)	NM STIMULUS
1	EXP change	M1_Coo	M1_Agg	Stim 1
2	EXP change	M2_Coo	M2_Agg	Stim 2
3	EXP change	M2_Agg	M2_Coo	Stim 3
4	EXP change	M1_Agg	M1_Coo	Stim 4
5	ID change	M2_Agg	M1_Agg	Stim 1
6	ID change	M1_Agg	M2_Agg	Stim 2
7	ID change	M1_Coo	M2_Coo	Stim 3
8	ID change	M2_Coo	M1_Coo	Stim 4

EXP: expression; ID: identity.

Figure 1.

Audiovisual Nonmatch-to-sample task. A) In the task subjects must press a button when a stimulus that does not match the presented sample is shown. Subjects fixated a center point to begin a trial and were then presented with a face and vocalization combination as the sample. On type 1 trials, a Nonmatch was shown immediately after the sample and a button press was required for juice reward. On type 2 trials, the sample stimulus was shown followed by a delay and then a repeat of the sample, (match stimulus). A second delay period ensued and then the third stimulus, the Nonmatch was shown and a button press was required for juice reward. B) The stimuli and contextual trial types. During Expression Change trials, the sample and Nonmatch would be from the same identity “actor” macaque but differ by expression. In contrast, during identity-change trials, the type of expression would be the same in the sample and the Nonmatch, but the identity was different.

Behavioral Task and Performance

Animals were trained to perform an audiovisual Nonmatch-to-sample (NMTS) discrimination task (Figure 1A), as has been previously described (Hwang and Romanski, 2015; Plakke et al., 2015). They were required to remember a face with an accompanying vocalization during the sample period and detect a nonmatching face and vocalization combination in subsequent stimulus presentations. The subject initiated a trial by fixating for 500 ms on a red square presented at the center of the screen. Then, the sample stimulus (i.e., the face+vocalization) was presented. The static face image and the vocalization both began at stimulus onset time 0 and the face remained on the screen for 1000 msec. After the sample period a delay period that varied between 1000 – 1250 ms followed. After the delay, either the sample would be presented again (Match stimulus) or a Nonmatching stimulus would be presented (Figure 1). The Nonmatching stimulus consisted of a face and vocalization combination that differed from the Sample, i.e. both the vocalization and the face stimulus differed in the Nonmatch in comparison to the sample (Figure 1). The subjects were required to detect the nonmatching stimulus, when it occurred, by responding with a button press to receive a juice reward. During stimulus presentations, the subjects were required to maintain their gaze within the viewing window corresponding to the visual stimulus on the monitor. If gaze was not maintained, the trial was aborted and re-started.

On type I trials, the Nonmatch test stimulus followed the sample after a 1000 – 1250 ms delay period. On type II trials, the sample was repeated as a matching stimulus, then a delay of 1000 – 1250 ms would occur and finally the Nonmatch stimulus would appear and the subject would press the button to detect it and receive a juice reward (Figure 1A). Trial types I and II were presented randomly, so that the subject could not predict the occurrence of the Nonmatch. All of the stimuli were congruent combinations (e.g., coo face + coo vocalization). The Nonmatch stimuli differed from the sample stimulus based on either a change in caller identity (face and vocalization from a different subject than that of the sample) or a change in emotional expression (different vocalization and facial expression but from the same individual as the sample) (Figure 1B). Within the testing/recording blocks, the sample and Nonmatch stimuli consisted of four face-vocalization stimuli: two different individuals (M1 = male, M2 = female) displaying two different face-vocalization calls as in Table 1: 1 positive/neutral expression (coo) and 1 negative expression (aggressive bark or scream). These stimuli were presented with a randomized balanced design that accounted for the different Nonmatch stimulus types: a change in identity using each type of emotional expression (male coo vs. female coo, male aggressive vs. female aggressive) and a change in emotional expression using each identity (male coo vs. male aggressive, female coo vs. female aggressive) (Figure 1 and Table 1). There were no trials that included both a change in expression and a change in identity. The experimental paradigm was also designed to account for presentation order (i.e. all stimuli were presented equal times as the Sample and the Nonmatch stimulus during the course of the 160 total trials in the training/recording block. Note that conditions 1–8, as shown in Table 1, were presented as both type I (Sample - Nonmatch) and type II (Sample – Match – Nonmatch) trials. As such, there were 16 different conditions during each testing block: 2 Trial types (Type 1 and Type 2); 2 trial context conditions (Expression change and Identity change) and 4 Nonmatch stimuli (2 actors making 2 expressions). With 16 total conditions at a minimum of 10 trials per condition, the total trial count goal was ~160 – 200 trials; the performance rate goal was 85% correct.

Both subjects were trained in the performance of the Nonmatch to sample task to a criterion of 85% correct performance before recordings commenced. Performance was calculated as a running total as animals are performing the task so that the number of trials presented (total trials) and the number of correct trials was continuously monitored. Large blocks of trials with long runs of incorrect trials were discouraged and resulted in time outs or a change in task parameters and remediation procedures to correct behavioral errors during training. During recording, several incorrect responses in a row would prompt a time-out or a cessation in data collection during recording. A recording session with insufficient trials in the task conditions was not counted toward the total cell count. While error trials are useful in assessing the neural basis of behavior, in this report, only the neurophysiological data recorded during correct trials are presented due to the low number of errors committed during recordings.

Recording Procedures

When training was completed, a titanium recording cylinder was placed over the VLPFC using MRI-guided coordinates, as previously described (Hwang and Romanski, 2015; Diehl and Romanski, 2014), centered 29–30 mm anterior to the interaural line and 20–21 mm lateral to the midline. The recording cylinder was angled ~ 27° to the vertical to maximize an orthogonal approach to VLPFC, areas 12/47 and 45 defined anatomically by Preuss and Goldman-Rakic (1991) and physiologically by Romanski and Goldman-Rakic (2002).

Recordings were made in both hemispheres of Monkey 1 (LS) and the left hemisphere of Monkey 1 (GT). During recordings, a single Parylene coated, tungsten electrode (impedance 0.7 – 2.0MOhms; FHC, Inc.) was lowered into the VLPFC each day, using an XY Narishige micromanipulator to allow for grid-position location information and alignment with post-perfusion histology. Single-unit activity was amplified, discriminated and recorded with a PLEXON MAP system while the monkey performed the Nonmatch to sample task. Stimulus display and task control was executed by NIH sponsored software, CORTEX and later, a custom program ORION. We recorded the activity of each stable unit that we encountered as the electrode was lowered into the brain. After a unit or cluster of units was recorded, the electrode was advanced by ~200 μm and a new recording block was begun.

At the end of the recording experiments, small electrolytic lesions were placed at predetermined locations within the chamber as X,Y registration marks or florescent tracers were placed into recorded regions and were used to define recording locations and afferent connections for additional studies. After 10 – 14 days, animals were euthanized and perfused, in accordance with the National Institutes of Health and the University of Rochester Care and Use of Animals in Research Committee. The brains were removed and processed histologically to reconstruct the electrode locations of the recordings. Nissl sections were used to plot and reconstruct the electrode tracks using NeuroLucida (MicroBrightfield Inc.). These were used to construct the cell response maps in Figure 8.

Figure 8.

Locations of recordings and responsive neurons for both subjects. The locations of the recording chambers (black circles) are shown for each subject on a whole brain schematic A) and a photomicrograph B) for each subject. The locations of recorded cells which had significant effects of identity (blue triangle), expression (red circle) or an interaction of expression*identity (purple square) are plotted in X, Y coordinates C) below each photomicrograph. Responsive neurons were frequently localized in the anterolateral quadrant of the recording chamber. The total area that was explored during recordings is shaded in gray.

Data Analysis

A total of 600 single-unit recordings was obtained from two animals. We removed any cells in which recordings were incomplete and any cells with a firing rate less than 0.5 Hz during the task period, which yielded 513 recorded units. We used a single response bin of 700 ms to assess activity in the Sample, Delay, and Nonmatch periods of the task based on pilot and previous analyses to accommodate stimulus durations and response latency and to avoid reward or button press response activity, if present. The stimulus response onset latency, calculated using the Poisson spike train analysis method (Romanski and Hwang, 2012), was used to determine the starting point of the response window. We classified units as task-related if they had a statistically significant change in neuronal activity in at least one of the analyzed task periods (Sample, Delay, Match or Nonmatch) compared to baseline activity during the inter-trial interval (Wilcoxon rank-sum test, p≤0.05). Task-responsive units were analyzed further for task-specific responses.

Nonmatch Period Analysis

In this task, the target stimulus that is presented after the delay will either match the presented sample stimulus (Match, trial type II) or was the target Nonmatch and differed from the sample stimulus (Nonmatch, trial type I). We investigated the effects of changes in expression or identity from the Sample to the Nonmatch stimulus on the neuronal activity during the Nonmatch period in type I trails. We have previously discussed that performance accuracy in type II trials (Figure 1) is not dependent on working memory (Hwang and Romanski, 2015) and may not reflect the same cognitive processes as the Nonmatch in Type 1 trials. We, therefore, focused the analysis in this report on the Nonmatch activity of only Type 1 trial.

We determined the start and ending points of our analysis bin by calculating the average onset response latency across all trials for each cell using the Poisson spike train analysis method as described previously (Romanski and Hwang, 2012). The starting point of the 700 ms bin was adjusted depending on each neuron’s average response latency. If response latency was less than 100 ms, the bin began at 0; if the response latency was 100 – 200 ms the bin started at 100 ms and if the latency was greater than 200 ms then the bin started at 200 ms post stimulus onset. Using this method, variations in latency with different stimuli on different trials could be accommodated.

The audiovisual NMTS task has eight conditions in which four face-vocalization stimuli (two monkeys each with two expressions) are presented as a sample or a Nonmatch in an identity-change trial or an expression-change trial (Table 1). Thus, the response may be a result of a specific stimulus presentation or may be a result of the specific contextual change that occurred. The same face will occur in the Nonmatch period under two different contexts – once in an identity-change trial (Sample and Nonmatch have the same expression but different identities) and once in an expression-change trial (Sample and Nonmatch are from the same identity monkey but have different expressions) (Table 1). Our goal was to determine if task context or identity and expression features of stimuli modulate neural activity in VLPFC neurons. We examined this with a two-way ANOVA with stimulus (4 levels, i.e. the 4 face-vocalizations that were the Nonmatch memoranda) and context (2 levels, i.e. identity or expression change trial) as the between-subjects factors. Planned comparisons were made between stimuli presented during the expression- or identity-change trials. We also assessed behavioral differences in identity- and expression-change trials. The reaction times in these two trial contexts were compared with a t-test in both subjects for each recording session.

In addition, we examined the response of VLPFC neurons to the Nonmatch stimulus regardless of trial context, since neurons may respond to only the stimulus that is presented on the screen at that time. We coded the Nonmatch stimuli by the identity of the face-vocalization stimuli (actor monkey 1 = M1; actor monkey 2 = M2) and the vocalization expression of each of the stimuli (affiliative coo and agonistic bark or scream) and conducted a two-way ANOVA (identity, Expression) on the mean firing rate during the Nonmatch stimulus period for each cell.

To assess whether expression- and identity-change trials differed in the timing of the neural response, we examined early and late Nonmatch responses in task-related cells by dividing the response window into two 400-ms bins. We performed a one-way repeated measures analysis with trial context (expression or identity change) as the grouping factor and the early and late stimulus period as the repeated measure.

RESULTS

In the current study we investigated how ventral prefrontal neurons respond to task and stimulus features, in particular, expression and identity information in audiovisual communication stimuli. Two rhesus macaques performed the NMTS task while neurophysiological recordings were made in VLPFC. The audiovisual social stimuli that were presented in the NMTS task differed according to the identity or the expression of the macaque face-vocalization stimulus presented. The stimulus was presented as the Sample, and, after a delay, either the same stimulus (Match) or a different stimulus (Nonmatch) was presented and subjects had to press a button to detect the Nonmatch stimulus when it occurred. The nonmatching stimulus differed by either identity (same facial expression and vocalization type) or by expression (same identity but different facial expression and vocalization type (Figure 1; Table 1).

Behavioral performance

During recording sessions, both subjects performed the NMTS task with an average accuracy that was ≥ 85% correct. Subject’s 1’s overall performance was 93.4% ± 7.9% over the course of 84 sessions and subject 2’s overall performance was 90.1% ± 6.7% over the course of 122 recording sessions. Reaction times, i.e., the times for correct button-press responses, were calculated from the onset of the audiovisual Nonmatch stimulus. Subject 1’s mean reaction times on type 1 and type 2 trials were 1405.91 ms 6.4 ms SEM ms and 1364.60 ms +/_ 12.8 ms SEM, respectively. Subject 2’s reaction times on type 1 and type 2 trials were 809.03 ms +/− 5.23 ms SEM and 783 ms +/− 8.2 ms SEM, respectively. As expected, both subjects were faster on type 2 trials (Subject 1, t (111) = 4.09, p < 0.001; Subject 2, t (142) = 5.87, p < 0.001) since the occurrence of the Nonmatch stimulus on type 2 trials is predictable and always occurs after the match stimulus (Subject 1,. The faster reaction time during this second Nonmatch has been observed in previous studies using this NMTS task (Hwang and Romanski, 2015; Plakke et al., 2015).

We also compared reaction time during trials where either expression or identity changed. Across 255 recording blocks, the average reaction time of subject 1 was 1409 ms +/− 4.2 ms SEM (range 1299 – 1645 ms) during expression trials and 1406 ms +/− 4.0 ms SEM (range 1296 – 1643 ms) during identity trials. Subject 2’s reaction times were shorter, with reaction times averaging 811 ms +/− 3.5 ms SEM; (range 635 – 946 ms) for expression trials and 813 ms +/− 3.6 ms SEM (range 634 – 943 ms) for identity trials across 282 recording sessions. We analyzed each recording session in the subjects separately and found that 16% of the recording blocks for subject 1 (23/143) but only 2% (2/112) of the blocks for subject 2 showed a significant difference between the reaction time of expression trials compared to identity trials (2 sample t-test, conducted for each cell, p ≤ 0.05). Subject 1 showed longer reaction times on expression trials than identity trials in 14/23 sessions. Subject 2 had longer reaction times on expression trials in one of the significant recording sessions compared to identity trials in that session.

We performed additional analyses to determine if specific stimuli used in the Nonmatch to sample task affected performance accuracy, since the stimuli differ by a number of features including spectral features, head tilt, sound duration, etc. In NMTS task there are four face+vocalization stimuli used as the memoranda in the task during the Sample, Match and Nonmatch epochs. Different face-vocalization exemplars were used to create multiple stimulus sets to be used in the task. There was a total of 4 sets used in training and three of these were used in the recordings for a total of 12 (3 sets with 4 stimuli each) face-vocalization stimuli used across all of the recordings performed. For each recorded cell, we performed a one-way ANOVA on performance accuracy (as percent correct) and grouped the cells by stimulus set for further analysis. There was no significant effect of stimulus on performance in the 3 stimulus sets used for the NMTS task: Set 1, F_3,148 = 0.4789, p = 0.6974; Set 2, F_3,220 = 0.8418, p = 0.4722; Set 3, F_3,160 = 0.6142, p = 0.6068. A box and whisker plot (Figure 2) displays the performance accuracy as percent correct which occurred during performance of the task for each of the 12 stimuli, grouped by stimulus set, across all recordings. An overall one-way ANOVA across the 12 stimuli was significant (F_{11, 528} = 17.73, p < 0.001) and indicated that the 4 stimuli from set 1 had a lower accuracy (average 0.83, range 0.81 – 0.84) than the stimuli from the other 2 stimulus sets. However, accuracy among these stimuli within set 1 was not significantly different, as stated above. Thus, the performance within the task for these stimuli did not differ across stimuli. This set was the first stimulus set used in the recordings for this task and performance increased in successive recordings over time.

Figure 2:

Performance accuracy across stimuli depicted with a bar and whisker plot. The performance accuracy (percent correct) is shown across all recordings for each of the 3 sets of stimuli (sets A, B and C) used in the audiovisual Nonmatch to sample task (n=12 stimuli total). In each stimulus set there were 4 stimuli, with 2 expressions (EXP 1 and EXP 2) produced by Monkey 1 (ID1) and by Monkey 2 (ID2). The middle line of the box represents the median and divides the data in half; the bottom of the box represents the median of the bottom half or first quartile and the top of the box the 3^rd quartile. The whiskers indicate the minimum and maximum values. Outliers exceed 1.5 x the interquartile range.

Task related neuronal activity

Out of 513 recorded single units (75%), 384 were defined as task-related since they demonstrated a significant change in neuronal activity in at least one of the task periods (Sample, Delay, Match, Nonmatch) compared to baseline activity (Wilcoxon rank-sum test, p≤0.05), (Figure 3). There were 165 units recorded in subject 1 and 219 units recorded in subject 2. Of the 384 unique task-related cells, 182 units (47%) demonstrated a significant change (p < 0.05) in firing rate in the Sample period compared to baseline; 204 (53%) during the Match period; 165 (43%) during the Delay period; and 269 (70%) during the Nonmatch period. Many neurons were active in two or more task epochs. A large proportion of the task-related cells had significant changes in firing rate during the Nonmatch period. Two example neurons are shown in Figure 3. The cell shown in Figure 3A (L174000 cell 1; b=G162000 cell 2) had significantly higher neural activity only in the nonmatch period compared to the sample, delay and match periods (Wilcoxon signed rank test, p< 0.001). In contrast, the cell shown in Figure 3B had increased firing during both the Sample and Nonmatch periods of the task (Wilcoxon signed rank test, Sample and Nonmatch p< 0.001), as did 36% (137/384) of neurons in this study which is typical of many stimulus-responsive VLPFC neurons (Hwang and Romanski, 2015).

Figure 3.

Task related Single Neuron Examples. Two task-related single cells with a significant increase in neural activity during the Nonmatch period of the task are shown across the 4 stimuli. A) The neural response across the 4 Nonmatch conditions for a single unit with selective activity during the Nonmatch period of the task is shown. The unit did not show any appreciable response during the sample or delay periods. B) A second unit which exhibited an increase in the neural response during the sample and the Nonmatch period is shown. For this neuron the response to the different 4 stimuli was not significantly different. For each panel the rasters for each trial are shown with a spike density function (SDF) just below. The X-axis is time (sec) during the trial and the Y-axis is the firing rate (spikes/second). The onset of the sample stimulus was at time 0.

The neuronal response latency during the Nonmatch presentation of the face-vocalization stimulus was calculated using the Poisson spike train analysis for all units that were responsive during the Nonmatch period according to the Wilcoxon rank sum test (n=269) (Romanski and Hwang, 2012). The neural response latency across all units during all type 1 Nonmatch trials was 212 ms ± 6.7 ms (mean ± SEM; the median was 200 ms). These values are similar to the onset response latencies observed previously for complex audiovisual stimuli presented during the Nonmatch to sample task (Romanski and Hwang, 2012).

Neuronal Responses to Trial Context and Stimulus Factors

The goal of these experiments was to determine if VLPFC neurons encode expression and identity features of face-vocalization stimuli and/or contextual task variables, in a working memory paradigm. During the NMTS task, subjects must detect the Nonmatch stimulus when presented. Thus, during the Nonmatch period, neurons might respond for a variety of reasons, including the type of contextual change that occurred in the Nonmatch period (identity change or expression change) or the specific face-vocalization stimulus (n=4) presented during the Nonmatch period, regardless of the context.

To determine if VLPFC neurons encoded stimulus or context, we performed a two-way ANOVA on each of the task-related neurons with stimulus (n = 4 face+vocalization combinations) and context (n = 2, expression-change or identity-change trials) as factors. There were 15% (58/384) of neurons with a main effect of stimulus (p < 0.05), with effect size being medium (η2 > 0.06) in 33/58 cells and large (partial η2 > 0.14) in 25/58 cells. Although there were many cells which had a change in firing during the Nonmatch period (n=269, stated above), there was a significant main effect of the trial context (expression- or identity- change trials) in only ~5% (18/384) of cells (p < 0.05), with medium effect size in 13/18 cells (η2 > 0.06). In addition, 5% (18/384) of cells had a significant interaction of stimulus and context (p < 0.05), with the effect of the interaction being large (η2 > 0.14) in 11/18 cells and medium (η2 > 0.06) in 7/18 cells. Cells with a significant effect of stimulus, but no significant interaction effect (n=52), exhibited responses to the Nonmatch stimuli that did not differ across expression or identity trial contexts in the task.

Planned comparisons were made using the estimated marginal means to assess the similarity in response between Nonmatch stimuli presented in the two different trial contexts (Figure 4A– J). The example cell in Figure 4A–B, had a significant effect of stimulus (F _{3, 78} = 4.144, p = 0.009, η² = 0.137). However, there was no effect of trial context (F_{1, 78} = 0.301, p = 0.585, η² = 0.004), and the response to each of the 4 stimuli (shown in colored SDF lines) did not differ during expression- and identity-change trials. For this cell, stimulus M2_AGG evoked the highest magnitude response, which was not significantly different in the expression- and identity-change trials (mean firing rate = 4.4 and 4.4 Hz, respectively; p = 0.852). The cell shown in Figure 4C, D also had a significant effect of stimulus (F_{3, 72} = 5.633, p = 0.002, η² = 0.190) with selectivity for stimulus M2_AGG, although this cell was tested with different exemplar stimuli than that of figure 4A,B. The effect of trial context was not significant (F_{1, 72} = 0.516, p = 0.475, η² = 0.007), and the response to all four stimuli during identity- and expression-change trials did not differ. In a third cell with a main effect of stimulus (F_{3, 88} = 9.460, p < 0.001, η² = 0.244) but no significant effect of trial context (F_{1, 88} = 1.872, p = 0.175, η² = 0.021; Figure 4E, F), the response to stimulus M1_Coo (light blue) was significantly increased compared to the other stimuli (p < 0.05, Tukey HSD) in both expression- and identity-change trials, which did not differ significantly (p = 0.287). Thus 15% (58/384) of neurons had a significant effect (p < 0.04) of the stimulus presented which was the same regardless of the context in which it was presented.

Figure 4.

The neural responses during the Nonmatch period of 6 neurons to the 4 stimuli during expression-change trials (left column) and identity-change trials (right columns) is shown as raster and SDF plots. The neurons depicted in panels A-B, C-D, E-F and G-H did not differ by trial context (expression- or identity-change trials) but differed according to the identity or expression of the stimuli themselves. The neuron in panels I-J exhibited a significant response during the Nonmatch period in expression-change trials (left column) but not in identity-change trials (right column) for the same Nonmatch stimuli. For each panel the rasters for each trial are shown with a spike density function (SDF) just below. The X-axis is time (seconds) during the trial and the Y-axis is the firing rate (spikes/ second). In this figure the onset of the Nonmatch stimulus was at time 0.

A final example of a neuron responsive to the Nonmatch stimuli and not the trial context is shown in Figure 4G, H. There was a significant main effect of stimulus (F=_{3, 72} = 12.613, p < 0.001, η² = 0.344) but not trial context (F=_{1, 72} = 1.473, p = 0.229, η² = 0.020) for the response of this neuron. The mean firing rate was significantly increased in two stimuli from the same identity, M2_coo, dark blue (p < 0.034) and M2_Agg, green (p < 0.042) compared to the other 2 stimuli, but did not differ by trial context (p = 0.52 and p = 0.16; assessed with planned contrasts of the estimated marginal means).

Only 4% (15/384) of cells demonstrated a significant main effect of context, where the response to the same Nonmatch stimulus differed across expression- and identity-change trials, indicating a comparison of the stimuli from sample to Nonmatch. In addition, 6% (18/378) of task-related neurons demonstrated a significant interaction (p < 0.05) of stimulus and context, i.e. the response of these neurons was dependent not only on the face-vocalization stimulus presented, but also on the particular context (expression- or identity-change trial) in which that stimulus was presented. The neuron shown in Figure 4I–J demonstrates this interaction of factors (context*stimulus, F_{3, 73} = 2.88, p=0.041, η² = 0.106) and the mean firing rate to both coo stimuli (dark blue and light blue) was significantly higher than the other stimuli, during expression-change trials but not during identity-change trials (M2_Coo p= 0.007 and M1_Coo p=0.016). Hence, these neurons potentially encode both stimulus and context information.

Nonmatch stimulus responses to identity and expression

Since many more neurons were responsive to the Nonmatch stimuli rather than the trial context, we examined the recorded population of cells (n = 513) to determine responsivity to the Nonmatch stimuli, regardless of task context. Here, the audiovisual stimuli which were presented as the sample, match or nonmatch stimuli were 2 identity monkeys performing 2 vocalization expressions. We analyzed neuronal firing during the Nonmatch stimulus using a two-way ANOVA on the stimulus factors of identity (n = 2 for the two different identities) and expression (n = 2 for the two different vocalization expressions). In 12% (63/513) of the total population of recorded cells, there was a significant effect of stimulus identity (p < 0.05), and in 9% (45/513), there was a significant effect of stimulus expression (p < 0.05) (Figure 5). Furthermore, 9% (45/513) of cells had a significant interaction of identity and expression (p < 0.05) and were most responsive to one of the four stimuli. In the raster-histograms of cells which were described in Figure 4, the neurons depicted which had an effect of stimulus can also be described in terms of their response to expression and identity of the nonmatch stimulus. Several of the neurons in Figure 4 A–F had a significant interaction of stimulus expression and identity, indicating that these neurons were significantly responsive to a particular stimulus during the task. In Figure 4A,B and 4C,D, stimulus M2_Agg elicited the highest response (Figure 4A,B, cell 2490001, Identity*Expression, F_{1, 82} = 11.9, p= 0.001, η² = 0.0497; Figure 4C,D cell 251000_2, Identity*Expression, F_{1, 76} = 6.05, P = 0.016, η² = 0.074). In contrast, in figure 4E,F the cell was selectively responsive (significant effect of Identity, F_{1, 92} = 7.5, p < 0.007, η² = 0.076) to stimulus M1_Coo (light blue) compared to the other stimuli (p < 0.01). The neuron illustrated in Figure 4G–H, had a main effect of identity (F_{1, 76} = 37.645, p < .001, η² = 0.331) and responded to both the AGG and COO stimuli from monkey identity M2 (green and dk blue lines). The neuron depicted in Figure 4 I–J was one of a few cells that exhibited a significant main effect of expression (F_{1, 77} = 5.584, p = 0.021, η² = 0.07), such that the M1 and M2 Coo stimuli (light blue and dark blue) elicited a significantly greater response compared to the M1 and M2 AGG expression stimuli.

Figure 5.

Identity and Expression stimulus responses. Mean firing rates of four neurons during the Nonmatch period of the NMTS task are shown in A-D, with the raster and spike density functions separated by identity (M1, monkey ID1; M2, monkey ID 2). Cells plotted in A and B had a main effect of identity with a significant increase in firing for identity 2 for cell A and identity 1 for cell B. In contrast, the cell depicted in C had a main effect of expression with a greater response to the affiliative expression (blue). The cell depicted in D had a main effect of both identity and expression, but also an interaction of expression and identity in the two-way ANOVA on the Nonmatch stimulus.

Additional examples demonstrating the selectivity of VLPFC neuronal responses for stimulus identity, expression or interaction, are portrayed in Figure 5 where the raster/spike density function plots have been separated by stimulus identity in the left and right columns. The responses for the two face-vocalization expressions from monkey 1 (M1) are shown in the left column and the two face-vocalization expressions from monkey 2 (M2) are in the right column of Figure 5. Cells A and B, had significant main effects of Identity. Cell A had a significantly increased response to the two face-vocalization expressions for the M2 identity (F_1,76 = 37.645, p < 0.001, η² = 0.33), and cell B had a significant difference in response which was greater for the M1 identity stimuli (F_{1, 76} = 7.03, p < 0.01, η² = 0.085).

Only 8% of the neurons investigated here (43/513) had a main effect of expression. The cell depicted in Figure 5C is an example of one cell that had only a main effect of expression (F_1, 76 = 24.306, p < 0.001, η² = 0.243) with a greater response to the affiliative coo-vocalization and face (blue line) compared to the aggressive face and vocal stimulus (red line) for both M1 and M2 identities. About 9% of neurons in our recorded population had a significant interaction of identity and expression. The cell portrayed in Figure 5D, had a significant main effect of identity (F_{1, 78} = 13.873, p < 0.001, η² = 0.151), wherein the response was greater for M2; a significant main effect for expression (F_{1, 78} = 14,718, p < 0.001, η² = 0.159) such that the response to the aggressive face-vocalization stimulus (red line) was greater than that for the affiliative (blue line), and a significant interaction of identity and expression (F_{1, 78} = 5.365, p < 0.01, η² = 0.064) illustrated by the fact that the response to the aggressive face-vocalization stimulus was enhanced for the M2 identity. There were 124 cells which exhibited an effect of identity, expression or an identity*expression interaction in the analysis of the Nonmatch stimulus response, with 37% (n=46/124) encoding only the identity of the Nonmatch stimulus (Figure 6).

Figure 6.

Venn Diagram of the population of cells responsive in a two-way ANOVA to the expression, identity or an interaction of these factors in the nonmatch stimulus. 45 cells had a significant interaction of expression and identity, while 63 cells had a significant effect of Identity and 43 cells, a significant effect of the expression of the nonmatch stimulus. These responses were not exclusive and many cells were responsive to the other factors as indicated.

Early and late Nonmatch response

Our results indicated that stimulus and, to a lesser extent, trial contextual factors contributed to the Nonmatch response in VLPFC neurons. Since identity and expression have been shown to recruit different neural circuits with potentially different response latencies, we asked whether there were temporal differences in the neural response during the Nonmatch period of expression and identity trial contexts. We, therefore, divided the Nonmatch period into an early (0 – 400 ms) and a late (401 – 801 ms) bin and conducted a one-way repeated measures ANOVA of trial context (expression- or identity-change trials) on early and late time bins of the Nonmatch period. Overall, 36/531 recorded cells had a significant interaction (p < 0.05) of trial context (expression- or identity-change trials) by response bin. During identity-change trials, more cells had a greater response during the early part of the Nonmatch period (n = 20) compared to the later part of the Nonmatch period (n = 14). In contrast, during expression-change trials, more cells demonstrated an increase in firing rate during the late part of the Nonmatch period (n=22) compared to the early part of the Nonmatch period (n = 16; Figure 7). However, there was not a statistically significant association between the two distributions (Fisher’s exact test p = 0.238, ns).

Figure 7.

Early and late responses in expression and identity trials. A subset of neurons had a significant effect of trial context, identity change or expression change trials. There was a general trend for more neurons to show a significant change in the early part of the Nonmatch period for identity change trials and the later part of the Nonmatch period for expression change trials.

Location of expression and identity related neurons in VLPFC

Histological examination of the location of the recording chambers and the recorded cells verified that both chambers targeted VLPFC (Figure 8). In subject 1, recordings were concentrated in the left VLPFC areas 45 and 46v. In subject 2, both the right and left recording chambers, included areas 12/47, 45 and 46v (Figure 8 A–B). Using the results of the two-way ANOVA on the Nonmatch period stimulus for identity and expression, cells that demonstrated a significant main effect of stimulus identity are shown as blue triangles, expression in red circles and a significant interaction of expression and identity are shown with purple squares. There was an overlap of expression-, identity- and expression*identity-related cells in the anterolateral quadrant of the recording chamber in both subjects. However, there were very few expression- or expression*identity-related cells in the posterior dorsal part of the chamber in the dorsal part of area 45, where face cells have been reported (O’Scalaidhe et al., 1999) and which appears coextensive with prefrontal face patch PA (Tsao et al., 2008). In both subjects, all responses were densest in the anterolateral part of the chamber, corresponding to the area 12 vl and the ventral part of area 45 (Preuss and Goldman-Rakic, 1991), in which cells have previously been shown to be vocalization responsive (Romanski and Goleman-Rakic, 2002) or multisensory (Sugihara et al., 2006; Romanski and Hwang, 2012; Diehl and Romanski, 2014). This anterolateral region may also be coextensive with face patch areas PL and part of the orbital patch PO (Tsao et al., 2008; Schwiedrzik et al., 2015).

DISCUSSION

VLPFC, a node of the social communication network, selectively responds to and integrates species-specific faces and vocalizations. In the present study, we have shown that VLPFC neurons responded to both task and stimulus factors, as subjects detected a change in identity or expression during a working memory task. VLPFC neurons had selective activity during particular task epochs, the most frequent being during the Nonmatch period, in which 70% of the task-related neurons were significantly responsive compared to baseline. A small number of task-related neurons was responsive to whether a trial involved a change in identity or a change in expression. In contrast, a large number of neurons responded to the stimulus that was presented during the Nonmatch period and specific identity or expression of the Nonmatch stimulus. Stimulus-responsive neurons were located in the anterolateral VLPFC region where multisensory and vocalization neurons have been previously described (Sugihara et al., 2006; Diehl and Romanski, 2014). Our results extend our understanding of the type of information that VLPFC neurons encode, to include expression and identity and to support the claim that the VLPFC is an important part of the social communication network.

Task-related responses

Given the high number of task-related neurons recorded in the present study, the audiovisual NMTS task appears to be an optimal way to investigate responses of VLPFC neurons to specific stimuli (Hwang and Romanski, 2015). Neurophysiological recordings during the audiovisual NMTS task in the present study indicated that 75% of the total recorded population (384/513 cells) was active during at least one epoch of the task including the sample, delay, match and Nonmatch periods. While about 50% of the task-responsive neurons were active during the sample or delay periods, activity during the Nonmatch period was more prevalent, with ~70% of task-related neurons demonstrating a significant change in firing rate during the Nonmatch, or decision period of the task, similar to previous studies (Hwang and Romanski, 2015). Since the Nonmatch period is when the discrimination of the memoranda and the decision to make a response both occur, performance accuracy may depend on neural activity in VLPFC during this epoch. This is suggested by the finding that inactivation of VLPFC with cortical cooling results in an impairment of response accuracy in the audiovisual NMTS task (Plakke et al., 2015).

One specific question we asked was whether single VLPFC neuronal activity represents task events, stimulus features or both. During this audiovisual NMTS task, the change from sample to Nonmatch involved a change in either identity or emotional expression of the face-vocalization stimulus that was presented in the Nonmatch period. Although a large portion of the recorded population was active during the Nonmatch period, only ~5% of cells encoded the contextual change that occurred during the trial, and only slightly more (6%) had a significant interaction of trial context and stimulus. In our task, the type of change that occurs in a trial is not an explicit factor, and the discrimination does not depend on choosing the specific change in the trial specified factor. Instead, during the NMTS task, subjects make a button press to detect the next stimulus that differs from the sample, regardless of whether the change is one in identity or expression. Thus, the low yield of neurons which encoded the contextual trial type may be due to the fact that this change was implicit. Nonetheless, single neuron examples indicate that, in some cells, responses during the Nonmatch period were context-dependent since the response to the same face-vocalization stimulus differed when it occurred in an identity-change trial versus an expression-change trial (Figure 4I–J).

We hypothesized that behavioral responses during expression-change trials might require more cognitive processing time and would result in changes in firing during the late part of the Nonmatch period. This is suggested by previous data from studies in human and nonhuman primates. Recordings from macaque VLFPC neurons found that information about the emotional expression of a face stimulus peaked later in neurons than the information about the identity in a face stimulus (Kuraoka et al., 2015). In human fMRI studies, expression recognition took longer in the superior temporal sulcus (STS), a temporal lobe cortical region with robust projections to VLPFC, than identity recognition (Pitcher, 2014). In our study, examination of the early and late responses during identity- and expression-change trials revealed a significant interaction between trial context and early or late response period in about 8% of the task-responsive neurons. Approximately 5% of those neurons were selectively responsive during the early part of the Nonmatch period in identity trials and a similar proportion of cells were selectively responsive during the late part of the Nonmatch period in expression trials (Figure 7). However, the difference between these two distributions was not significant and the question as to whether additional time is needed to process expression versus identity information in VLPFC neurons merits further study.

Identity and Expression Neurons in VLPFC

Responses during the Nonmatch period may be due to task processes (as discussed above) or to the sensory features of the face-vocalization stimulus itself. In 24% of the total recorded population (124/513) of VLPFC neurons, the change in firing rate was related to the identity and/or expression of the face-vocalization stimulus itself. Neurons which encoded identity accounted for 12% of the recorded population while expression neurons accounted for < 8%. These small percentages may be because each of the stimulus sets feature only two “actors” with two expressions. With regard to the greater number of identity-encoding cells, the number of features that contribute to distinguishing two identities, and which may drive additional cells, may be greater than the number of features that differ between the two expressions of a given actor. The representation of identity and expression in VLPFC neurons may be further complicated by whether a given cell integrates face and vocal information. Neurons that process non-changeable features like identity may rely mostly on visual cues and the number of visual responsive neurons is greater in VLPFC than auditory responses (Romanski and Goldman-Rakic, 2002). In contrast, for the processing of expressions, it is well known that there are robust changes in the acoustic information across the different vocalizations which accompany the changeable features in facial expressions of macaques (Partan, 2002). Thus, expression discrimination may require integration of both face and vocalization information, which occurs in particular regions of VLPFC.

Neurons which encoded identity were found over a large region of the VLPFC recording area including the posterior and dorsal part of the recording region closest to the arcuate sulcus (area 45) and most prominently, in the anterolateral part of the recording region (area /12) (Figure 8). In contrast, expression-encoding cells and cells which encoded both expression and identity were mostly limited to the anterolateral portion of the recorded region (Figure 8). This anterolateral region, which corresponds to area 12 (Preuss and Goldman-Rakic, 1991), is also the region in which vocalization-responsive neurons are localized (Romanski and Goldman-Rakic, 2002; Romanski et al., 2005) and where neurons that integrate faces and vocalizations are most prevalent (Sugihara et al., 2006; Diehl and Romanski, 2014). This area also receives afferents from the basal nucleus of the amygdala (Sharma et al., 2020). These facts suggest a dissociation within VLPFC of identity and expression specialization in posterior and anterior regions, respectively. Furthermore, it suggests that the encoding of expression may require the addition of vocal information. In contrast, the processing of identity might proceed with face information alone. For example, it has been suggested that there is a “face” advantage over voices (Barsics, 2014; Stevenage et al., 2012) and that identity and other information may be processed in separate pathways. It has been suggested that emotional expression, may be extracted from voices in a separate neuronal pathway (Stevenage et al., 2012). The convergence of these separate face and vocal information pathways in VLPFC may provide the substrate for the encoding of both identity and expression in VLPFC neurons. However, these complex features may require processing by ensembles of neurons rather than single cells as shown by the small subsets of responsive neurons in the present study.

One shortcoming of the present study is the lack of information regarding whether expression- or identity-encoding cells were multisensory. To determine whether neurons that respond to vocalization or expression are predominantly multisensory it is necessary to record from a larger population of cells that can be extensively tested with multiple exemplars of faces and vocalization types across modalities. Recordings using implanted chronic arrays have the advantage of allowing for extensive testing of multiple stimulus sets across many cells simultaneously and over successive recording days. This strategy would be ideal in examining stimulus features, modality specificity and task-related processing across VLPFC.

Few studies have directly examined encoding of expression and identity in prefrontal neurons. A previous study that localized face patches to the prefrontal cortex suggested that the orbitofrontal face patch, PO, was activated by expressive faces (Tsao et al., 2008). A second study examined single neuron responses to dynamic face stimuli of different expressions and identities in VLPFC and the amygdala (Kuraoka et al., 2015). Similar to our study, there was selectivity for identity, expression, or both in some single VLPFC neurons. The amount of information in VLPFC neurons was assessed and, surprisingly, it was found that more VLPFC neurons had greater information about expression compared to identity (Kuraoka et al., 2015). Furthermore, the facial expression that preferentially activated VLPFC neurons was the coo expression. This is interesting given that coo vocalizations are quite variable and are used in a variety of social situations. Hence, they have been described as more “communicative” in nature rather than emotional. Acoustic exemplars of coo vocalizations were also found to carry more information than other acoustic vocalizations in the macaque repertoire in recordings of single VLFPC neurons (Plakke and Romanski, 2016).

Using larger stimulus sets and passive viewing, previous studies have described selectivity for particular faces, face-views, objects and vocalizations in VLPFC neurons (O’Scalaidhe et al., 1997, 1999; Romanski et al., 2005; Romanski and Diehl, 2011; Plakke and Romanski, 2016; Constantinidis and Qi, 2018). While the current recordings were conducted using a cognitive task that engaged subjects and controlled the allocation of attention to the task, the number of stimuli in a given testing set was only 4. The small number of stimuli made it difficult to examine the effect of stimulus features in VLPFC neurons. Testing with larger stimulus sets that are carefully selected for specific features or categories similar to the studies carried out in the temporal lobe face-patch system (Chang and Tsao, 2017) may help to pinpoint common features that account for selectivity in VLPFC.

Functional connections of VLPFC involved in face-vocalization processing

Information regarding identity and expression are likely transmitted to VLPFC neurons through afferents from the temporal cortex and amygdala. In nonhuman primates there are six discrete face processing areas of the temporal cortex where neurons have higher firing rates to images of faces compared with other objects (Tsao et al., 2006). View or pose-invariant face identity information is more prevalent in the anterior patches, while information about the orientation of the head is increased in more posterior face patches. Thus, while cells in face patches ML/MF are tuned to specific face views, cells in the more anterior AM face patch code view-invariant identity (Freiwald and Tsao, 2010). There is also a trend for medial face patches to be sensitive to facial motion and facial expression, which are changeable features (Haxby et al, 2002). A recent fMRI investigation showed that the face-selective patches in the STS fundus, AF and MF, were most sensitive to facial expression as was the amygdala, whereas those on the lower, lateral edge of the sulcus, AL and ML, were more sensitive to head orientation (Taubert et al., 2020). Face patch AF has also been shown to be multisensory and is responsive to species-specific vocalizations and corresponding dynamic facial expressions (Khandhadia et al., 2021). Thus, face patch AF, which is sensitive to motion (Furl et al., 2012), facial expressions and responds to both vocalizations and corresponding facial gestures, has much in common with VLPFC neurons and may be the primary source of afferents to the VLPFC from the macaque temporal lobe face patch network. Direct evidence for an AF to VLPFC projection is lacking. Analysis of projections directly from other face patches AL, ML, and MF, with fMRI guided anatomical tracing, did not reveal strong projections to VLPFC (Grimaldi, et al., 2016). However, VLPFC face and vocalization- responsive regions receive robust projections from neurons in the dorsal bank of the STS, fundus and regions of IT, which may be coextensive with several face patches including AF as well as multisensory area TPO (Diehl, et al., 2008).

Single cells in the lateral, basal and accessory basal nuclei of the amygdala have also been shown to respond selectively to face-identity and emotional expression (Gothard et al., 2007; Hoffman, et al., 2007). While some single units were significantly responsive to identity or expression, the number of cells that showed significant interactions or a main effect of both categories of stimuli exceeded the number of cells selective for either identity or expression (Gothard et al., 2007). Further studies suggest that the basolateral amygdala is multidimensional, much like the VLPFC, with complex response properties that encode features of social stimuli and contextual elements that may guide flexible responses. Because dense projections from the basal nucleus target VLPFC (Porrino et al., 1981; Barbas and De Olmos, 1990), it is likely that these identity- and emotion-responsive neurons in the basal nucleus may target the identity and emotion-responsive neurons in VLPFC. Physiologically guided injections of tracers directly into VLPFC face and vocalization responsive regions labels the amygdala, particularly the intermediate subdivision of the basal amygdala nucleus (Sharma et al., 2020), which may provide social stimulus related information to VLFPC. Simultaneous, dual recordings are needed to determine what information is sent from temporal lobe face processing areas to the VLPFC and how that information is utilized.

Mixed Selectivity in VLPFC

Here, VLPFC neurons were responsive to both task and stimulus factors, exhibiting what has been termed “mixed selectivity”. This term is described in studies where the firing rate of prefrontal neurons represented multiple task variables or stimulus features (Warden and Miller, 2010; Rigotti et al., 2013). In the current study, 70% of the recorded population had a significant change in firing rate during at least one epoch of the task and 75% of task-related cells were significantly responsive during the Nonmatch/decision period of the task. Approximately 24% of these task related cells exhibited a change in firing rate that was related to features of the stimuli presented during the Nonmatch period. This joint processing of task and stimulus attributes has been demonstrated in previous studies of dorsolateral (DLPFC) and VLPFC, where the features of sensory cues and task variables jointly drive prefrontal neurons (Takeda and Funihashi, 2002; Chafee and Goldman-Rakic, 1998; Dang et al., 2021; Fuster, et al., 2000; Warden and Miller, 2010; Hwang and Romanski, 2015). Hence, mixed selectivity is likely a general feature of prefrontal neurons, which receive afferents from sensory, motor and association cortices and integrates them for upcoming task demands and contextual states to guide behavior. Thus, multisensory integration, sensori-motor integration or multimodal responses to task and stimulus variables is to be expected in this integrative cortical region. The extent to which different regions of prefrontal cortex encode specific stimulus features (e.g., color, shape, pitch, identity, expression, etc.) versus particular task variables or other features will differ depending on the amount and type of sensory information that targets each cortical region and the current task or goal. In VLPFC, afferents from the temporal lobe provide the substrate for sensory responses to both face and vocal stimuli, which are recognized, discriminated, and remembered during social communication.