Reading Affect in the Face and Voice

Abstract

Context

Understanding a speaker’s communicative intent in everyday interactions is likely to draw on cues such as facial expression and tone of voice. Prior research has shown that individuals with autism spectrum disorders (ASD) show reduced activity in brain regions that respond selectively to the face and voice. However, there is also evidence that activity in key regions can be increased if task demands allow for explicit processing of emotion.

Objectives

To examine the neural circuitry underlying impairments in interpreting communicative intentions in ASD using irony comprehension as a test case, and to determine whether explicit instructions to attend to facial expression and tone of voice will elicit more normative patterns of brain activity.

Design, Setting, and Participants

Eighteen boys with ASD (aged 7–17 years, full-scale IQ >70) and 18 typically developing (TD) boys underwent functional magnetic resonance imaging at the Ahmanson-Lovelace Brain Mapping Center, University of California, Los Angeles.

Main Outcome Measures

Blood oxygenation level– dependent brain activity during the presentation of short scenarios involving irony. Behavioral performance (accuracy and response time) was also recorded.

Results

Reduced activity in the medial prefrontal cortex and right superior temporal gyrus was observed in children with ASD relative to TD children during the perception of potentially ironic vs control scenarios. Importantly, a significant group X condition interaction in the medial prefrontal cortex showed that activity was modulated by explicit instructions to attend to facial expression and tone of voice only in the ASD group. Finally, medial prefrontal cortex activity was inversely related to symptom severity in children with ASD such that children with greater social impairment showed less activity in this region.

Conclusions

Explicit instructions to attend to facial expression and tone of voice can elicit increased activity in the medial prefrontal cortex, part of a network important for understanding the intentions of others, in children with ASD. These findings suggest a strategy for future intervention research.

Impairments in social communication are core features of autism spectrum disorders (ASD). Even high-functioning individuals with ASD who have advanced formal language skills (ie, phonology, morphology, and syntax) are impaired in pragmatics (ie, the social use of language in context).¹ Understanding the communicative intentions of others is particularly difficult for individuals with ASD when nonliteral language is used, as in the case of irony.^2–5

In everyday social interactions, appreciating irony is likely to require both attending to the speaker’s facial expression and tone of voice and integrating the information gleaned from these cues with the context of the situation at hand. Typically developing (TD) children are sensitive to affect conveyed through the human face and voice from very early on.⁶ Individuals with ASD, however, do not show the same early preference for faces^7,8 and voices^9,10 and often have difficulty extracting the affect expressed through these cues.^11–15 Accordingly, neuroimaging studies have found that individuals with ASD show reduced activity in brain regions that respond selectively to the face and voice. Specifically, children^16–18 and adults^19–23 with ASD typically exhibit hypoactivation of the lateral fusiform gyrus (FG), the so-called fusiform face area, when viewing faces and facial emotions. With regard to voice perception, adults with autism fail to show voice-selective activity in the superior temporal sulcus despite a normal response to nonvocal sounds.²⁴ Other studies of higher-level abilities that build on face and voice perception, such as language and “theory of mind,” have shown reduced activity in prefrontal regions, ineluding the left inferior frontal gyrus^25,26 and medial prefrontal cortex (MPFC).^25,27–29

Evidence is emerging, however, to suggest that individuals with ASD can show more typical levels of neural activity in regions important for processing facial and vocal emotions if task demands allow for more cognitive or explicit processing. In a prior functional magnetic resonance imaging study, we asked children and adolescents with ASD to perform 2 tasks.^16,17 In the matching task, participants selected 1 of 2 facial expressions to match the emotion shown in a target face. In the labeling task, they picked 1 of 2 words to describe the target expression. The ASD group showed reduced activity in the FG relative to TD controls during the more automatic matching task but not during the more cognitive or explicit labeling task. Similarly, Critchley et al¹⁹ observed hypoactivation of the amygdala when adults with autism labeled the gender of emotional faces (implicit condition) but comparable activity when labeling the emotion in the same faces (explicit condition). In the domain of language, children with ASD activated right frontotemporal regions typically associated with prosodic processing^30,31 when attention to intonational cues was integral to the task.⁵ However, when automatic processing of prosody was required, individuals with ASD showed abnormal electrophysiological responses relative to controls.^32,33

Thus, there is some evidence to indicate that individuals with ASD may engage more normative neural networks when task demands require the explicit processing of emotion. However, in previous studies, facial and prosodic cues were presented in isolation. Little is known about the use of these cues in interpreting the communicative intent of others. We recently examined the role of prosody in understanding irony and found that individuals with ASD did activate regions recruited by TD controls when task demands required them to rely on the strong intonational cues provided.⁵ However, understanding the mental state of a speaker in a conversational setting typically also involves drawing on the important information conveyed through facial expression. As the number and complexity of available cues increases, children with ASD have been observed to make more errors in social perception, perhaps due to deficits in attention.³⁴ Indeed, individuals with autism have demonstrated abnormal visual fixation patterns when viewing social scenes.³⁵

The goals of this study were thus 2-fold. First, we sought to examine the neural correlates of inferring the communicative intent behind a potentially ironic remark in children with ASD using a naturalistic context where facial, prosodic, and contextual cues were available. Second, and perhaps most importantly, we examined whether explicit instructions to attend to facial expression and tone of voice would elicit more normative activation patterns in children and adolescents with ASD.

Based on the results of a recent study using the same paradigm with TD children and adults,³⁶ we expected scenarios involving irony detection to elicit greater activity in the MPFC and superior temporal gyrus (STG) than scenarios containing only literal utterances in TD children. Given behavioral research indicating that individuals with ASD are more likely than controls to interpret an ironic utterance literally,³⁷ we expected less selective activity for potentially ironic scenarios compared with unambiguous scenarios in the ASD group. We further predicted that explicit directions to attend to important social cues would yield increased activity in regions important for processing emotion or intent in the ASD group. In contrast, the TD group should exhibit high levels of activity in these networks regardless of task instructions. Based on previous research described earlier, candidate regions for showing the predicted group × condition interaction include the MPFC, FG, amygdala, and frontotemporal networks.

METHODS

PARTICIPANTS

The ASD group comprised 18 right-handed boys with a mean (SD) chronological age of 12.5 (3.0) years (range, 7–17 years) and a mean (SD) receptive language age of 13.4 (4.9) years (range, 7–22 years) measured by the Peabody Picture Vocabulary Test, third edition.³⁸ Participants were recruited through the Autism Evaluation Clinic at the University of California, Los Angeles and flyers posted at regional centers throughout Los Angeles. A diagnosis of ASD was established using the Autism Diagnostic Observation Schedule³⁹ and the Autism Diagnostic Interview–Revised⁴⁰ and supported by expert clinical opinion following DSM-IV criteria. We used the Social Responsiveness Scale⁴¹ to evaluate the severity of social impairment, although this information was not obtained for 3 participants. All of the subjects had fluent language skills (based on parental report and direct observation during the Autism Diagnostic Observation Schedule) and a full-scale IQ higher than 70 based on the Wechsler Abbreviated Scale of Intelligence,⁴² which has recently been shown to have excellent predictive accuracy for high-functioning individuals with autism, even for those with an atypical subtest profile.⁴³ Participants had no reported history of neurological disorders (eg, epilepsy), psychiatric disorders other than autism (eg. schizophrenia, attention-deficit/hyperactivity disorder [ADHD]), or structural brain abnormalities.

For the comparison group, 18 right-handed TD boys (mean [SD] age, 11.8 [1.9] years; range, 9–15 years) were recruited from the community through flyers posted at the university and in the Los Angeles area. None had a history of head trauma or medical, neurological, or psychiatric disorders, including ADHD, according to parental report. The Social Communication Questionnaire⁴⁴ was used to screen for the presence of autistic symptoms.

The ASD and TD groups did not differ significantly in chronological age, performance IQ, or full-scale IQ (Table 1), although the mean verbal IQ (VIQ) was higher in TD children than in children with ASD (t_1,34=2.9, P=.007, Cohen d>0.90; for all other comparisons, P>. 10). Seven participants with ASD were receiving psychotropic medications (including selective serotonin reuptake inhibitors, stimulants, and new-generation neuroleptics). We performed all of the analyses without these subjects and found that the pattern of results remained the same. As such, we report the findings with the full sample of 18 subjects in each group. Participants and their parents gave written informed consent to participate in the study according to the guidelines of the institutional review board. Subjects were paid $25 per hour (up to $75) for participating.

Table 1.

Subject Characteristics

Characteristic	TD Group Mean (SD)	ASD Group Mean (SD)
Age, y	11.8(1.9)	12.4(2.9)
VIQ	110(15)	94(18)
PIQ	105(18)	106(19)
FSIQ	108(17)	98(17)
SRS score	NA	99 (24)

Abbreviations: ASD, autism spectrum disorders; FSIQ, full-scale IQ; NA, not applicable; PIQ, performance IQ; SRS, Social Responsiveness Scale: TD, typical development; VIQ, verbal IQ.

STIMULI

Participants viewed cartoon drawings of children in conversational settings while listening to short vignettes ending with a potentially ironic remark (Figure 1). Each scenario had an ironic version and a sincere version that shared the same neutral setup. The ironic ending contained an undesirable outcome and a final remark uttered in a clearly ironic tone of voice. The sincere ending had a positive outcome with a final comment made in a sincere, complimentary tone of voice. Following the sincere or ironic comment, participants decided whether the speaker really meant what he or she said. Yes or no judgments were indicated on a handheld response pad. Instructions were clear that a yes response should be given for a sincere comment that should be taken literally, whereas a no response should indicate an ironic remark that meant the opposite of what was said. Participants were shown examples of sincere and ironic scenarios not used during the scan and all answered correctly.

Figure 1.

Example scenario. The setup (top) is shared by both the sincere and ironic versions of the scenario. The sincere ending is shown at the bottom left, and the ironic ending is displayed at the bottom right. The text below the drawings represents the accompanying auditory stimuli. Participants view the setup first, then either the sincere or ironic version of a scenario, followed by a blank screen and the question, “Did Ed mean what he said?”

To verify that the final comments sounded sincere or ironic as intended, 12 adult volunteers listened to the remarks presented without the surrounding context and rated them on a scale of 1 to 7 (1 was the anchor for ironic and 7 for sincere or complimentary). Mean (SD) ratings were 1.4 (0.7) for ironic remarks and 6.6 (0.7) for sincere comments. Sincere and ironic versions of each scenario were matched in terms of syntactic structure, semantic complexity, and length. To examine the neural circuitry underlying the interpretation of irony per se, we also included a no irony condition comprising scenarios ending in straightforward, unambiguous remarks (eg, “Ashley and Zack are riding their bikes. When it starts to get dark out, Ashley says, ‘Let’s go home.’”) These remarks were not easily interpretable in a non-literal light and were made in a neutral tone of voice. All of the scenarios were tested previously in 12 normal adults and 12 TD children in a study of developmental change in the neural basis of interpreting communicative intent.³⁶

ACTIVATION PARADIGM

Four activation blocks were interspersed with 5 rest periods. The first 3 blocks each contained 6 different scenarios (3 ironic, 3 sincere) ending in a potentially ironic remark (eg, “Thanks a lot!”). Instructions given before the first block were simply to pay attention (neutral instructions condition). Before the second and third blocks, participants were told to pay attention to the facial expression (attend face condition) or to the tone of voice (attend prosody condition). We chose to put the neutral instructions condition first to be able to examine participants’ natural response to potentially ironic scenarios without any carryover effect of instructions to attend to a specific cue. The order of instructions to attend to the face or voice in the second and third blocks was counterbalanced across subjects. The fourth and last activation block comprised 6 no irony control scenarios ending in an unambiguous statement (eg, “Please pass the crayons.”). To avoid any specific item effects, scenarios were used equally often in each of the 3 irony conditions across subjects. Furthermore, participants saw only 1 version (sincere or ironic) of each scenario. There were 18 potentially ironic and 6 control scenarios in all; each lasted 15 seconds. Activation blocks were interspersed with 21-second rest periods for a total length of 7 minutes 45 seconds. Response times and accuracy were recorded during scanning.

DATA ACQUISITION

Images were acquired on an Allegra 3-T scanner (Siemens Medical Systems, Malvern, Pa). A T2-weighted sagittal scout was used to prescribe the planes of the functional images and rule out structural abnormalities. For each subject, the functional data were 155 whole-brain axial volumes collected parallel to the anterior-posterior commissure line using a gradient-echo, echoplanar imaging sequence (repetition time, 3000 ms; echo time. 25 ms; 3-mm section thickness with 1-mm gap; 64 × 64 matrix size; field of view, 20 cm). A coplanar, high-resolution structural volume was also acquired using echo-planar imaging (repetition time, 5000 ms; echo time, 33 ms; 128 × 128 matrix size; field of view, 20 cm). No between-group differences were observed in mean head motion detected during the functional scan.

DATA ANALYSIS

Imaging data were analyzed using SPM99 (Wellcome Department of Cognitive Neurology, London, England; http://www.fil.ion.ucl.ac.uk/spm). Functional images were realigned to correct for head motion, spatially transformed into a Talairach-compatible atlas for intersubject averaging,⁴⁵ and smoothed (6-mm full-width half-maximum isotropic gaussian kernel) using Automated Image Registration.⁴⁶ For each subject, condition effects were estimated according to the general linear model using a 6-second delayed boxcar reference function. Response time and accuracy scores collected during scanning were entered as regressors to ensure that differential activation observed between conditions or groups were not due to differences in task difficulty. The resulting contrast images were entered into group analyses using a random-effects model to allow for population-level inferences.⁴⁷ Results were initially explored using liberal thresholds of P<.05 uncorrected for multiple comparisons for both magnitude and spatial extent. However, we consider significant and discuss only those activations that survived a more stringent extent threshold of P<.05 corrected for multiple comparisons at the cluster level (at least 115 contiguous voxels) with peaks of regional activity having a magnitude of t greater than 2.57 at the voxel level. For each group, 1-sample t tests were conducted to identify clusters of significant activity for each condition. Between-group differences were examined using 2-sample t tests within regions where significant activation was detected in either group across all conditions. This was accomplished by creating an image file for each group consisting of significant activity across all activation conditions vs rest and combining these images to create an inclusive mask.

Because verbal ability was higher in the TD group than in the ASD group, we used analyses of covariance to confirm that any between-group differences observed were not driven by differences in VIQ. For children with ASD, a multiple regression was conducted to identify regional activity associated with symptom severity controlling for VIQ. Based on previous research, regions implicated in reasoning about others’ intentions (ie, MPFC. superior temporal sulcus, and temporal poles) and processing facial expression (ie, FG, amygdala, and superior temporal sulcus) and tone of voice (frontotemporal networks) were considered regions of interest. Within these regions for which we had a priori hypotheses, we used a small volume correction to test for significance using a sphere with an 8-mm radius centered at the local maxima (corresponding to a minimum of 55 contiguous voxels).⁴⁸ To assess the robustness of our findings, effect sizes were computed for all of the significant between-group differences. All of the effect sizes exceeded the traditional benchmark qualifying large effect (ie, Cohen d>0.80).

RESULTS

BEHAVIORAL RESULTS

No significant between-group differences were observed for response time or accuracy, defined as the proportion of correct responses divided by total responses, either overall or in any of the conditions (Table 2) (P>.10; data unavailable for 1 participant with ASD). However, across all irony conditions, children with ASD answered significantly fewer questions correctly than children in the TD group (ASD group: mean [SD] number of correct answers, 16.1 [1.0]; median, 16; TD group: mean [SD] number of correct answers, 17.1 [1.4]; median, 18; F_1,33=6.95, P = .01). This reflects the fact that children with ASD had a significantly greater number of no responses than TD children (t₃₃=2.71, P = .01). Although participants in the ASD group did not differ from controls in the accuracy of their judgments when they responded, the greater number of no responses may reflect more difficulty inferring the communicative intent of the speaker.

Table 2.

Behavioral Performance

	Accuracy, Mean (SD) [Median], % Correct		Response Time, Mean (SD), s
Condition	TD Group (n = 18)	ASD Group (n = 17)	TD Group (n = 18)	ASD Group (n = 17)
Neutral instructions	95.2 (9.8) [100]	95.7 (8.1) [100]	2.46(0.22)	2.50 (0.57)
Attend face	96.3 (12.2) [100]	93.9 (8.5) [100]	2.42 (0.37)	2.55 (0.77)
Attend voice	98.0 (6.0) [100]	92.9 (12.1) [100]	2.41 (0.41)	2.47 (0.63)
No irony control	99.1 (3.9) [100]	97.9 (5.7) [100]	2.55 (0.33)	2.70 (0.62)

Abbreviations: ASD, autism spectrum disorders; TD, typical development.

Accuracy was positively correlated with VIQ in both the ASD (r₁₅=0.51, P = .04) and TD (r₁₆=0.48, P=.04) groups. Because the number of correct responses differed significantly between the groups, we used regression analyses to probe for a relationship between this variable and brain activity and found no significant correlations in any of our regions of interest.

FUNCTIONAL MAGNETIC RESONANCE IMAGING RESULTS

Effect of Irony

To examine the networks specific to processing irony, we compared the activity summed across the 3 irony conditions (neutral instructions, attend to face, attend to prosody) with the no irony condition within each group. As expected, TD children showed significant activity bilaterally in the STG and in the dorsal and ventral MPFC. In contrast, despite showing similar activity in primary auditory and visual cortices during the no irony condition compared with TD children, participants with ASD did not recruit any brain regions during the irony conditions beyond those engaged during the no irony condition, where the speaker’s intent was unambiguous.

A less selective neural response to ironic scenarios relative to nonironic scenarios in the ASD group was confirmed using a 2-sample t test. Activity in bilateral temporal regions and the MPFC was indeed greater in TD children than in children with ASD for the irony vs no irony comparison. An analysis of covariance confirmed that the TD group showed a stronger differential response to ironic scenarios in the MPFC and right STG, although between-group differences in the left temporal regions were no longer significant after controlling for VIQ (Table 3).

Table 3.

Peaks of Activation for the All Irony vs No Irony Comparison^*

			TD Group				TD>ASD				ASD>TD
Anatomical Region	BA	H	x	y	z	t Score	x	y	z	t Score	x	y	z	t Score
Anterior cingulate	32	R	4	42	12	3.92
Medial prefrontal cortex	9						−8	50	30	3.33†
	10	L	−6	64	12	3.18	0	52	28	2.84†
	10	R	14	50	−4	3.31
Superior temporal gyrus	42	L	−58	−24	6	5.16
	42	R	48	−21	10	4.89‡
	22	L	−64	−20	2	5.94	−64	−28	0	2.94
	22	R	56	−10	0	5.23‡	50	2	−4	3.40†
Angular gyrus	39	R									40	−64	20	3.29
Middle temporal gyrus	21	L					−56	−36	0	2.94

Abbreviations: ASD, autism spectrum disorders; BA, Brodmann area; H, hemisphere; L, left; R, right; TD, typical development.

^*Thresholds were P<.05 corrected for multiple comparisons at the cluster level and P<.05 at the voxel level (t>2.57 at peak).

^†Significant after covarying out verbal IQ.

^‡Survive only a small volume correction for multiple comparisons at P<.05 (k≥55 voxels)

Effect of Attentional Modulation

In the neutral instructions condition, interpreting a speaker’s communicative intent was associated with activity in temporal and occipital cortices bilaterally in both groups. However, TD children also recruited prefrontal regions, including the MPFC and inferior frontal gyrus bilaterally. In contrast, participants with ASD did not show any frontal activity when instructions were neutral (Table 4 and Figure 2A), even when the spatial extent threshold was lowered to P<. 05 uncorrected for multiple comparisons.

Table 4.

Peaks of Activation During Irony Conditions^*

			Neutral Instructions vs Rest								Attend to Face or Prosody vs Rest
			TD Group				ASD Group				TD Group				ASD Group
Anatomical Region	Putative BA	H	x	y	z	t Score	x	y	z	t Score	x	y	z	t Score	x	y	z	t Score
Medial prefrontal cortex	8	L									−6	40	48	4.41
	9	L	−10	58	28	4.92					−6	52	30	3.97	−8	52	32	3.35
	9	R	2	50	20	3.85
	10	L	−2	62	18	7.44					0	62	20	6.69
	10	R													4	58	14	3.13
Precentral gyrus	6	R	48	2	34	3.80									34	2	36	4.34
Middle frontal gyrus	9	R													40	12	38	3.64
	46	L									−44	42	4	3.29
Inferior frontal gyrus	44	L	−58	18	16	3.23					−56	16	14	3.59	−50	14	26	3.25
	44	R	32	14	22	3.79†					50	18	24	4.19†	52	16	16	3.93
	45	L	−56	22	20	2.91					−56	18	8	5.67	−52	20	18	3.35
	45	R	50	24	10	5.30†					50	24	16	5.51†	40	30	0	5.27
	47	L	−44	22	−10	4.73					−48	30	−2	6.16
Transverse temporal gyrus	41/42	L	−48	−30	8	10.11	−66	−32	10	6.41	−52	−28	6	10.99	−40	−32	14	3.38
	41/42	R	48	−24	6	9.06	52	−26	10	6.10	52	−20	10	7.65	54	−24	6	6.48
Superior temporal gyrus	22	L	−60	−22	2	10.61	−56	−30	2	8.01	−58	−22	4	16.63	−54	−20	4	6.31
	22	R	54	−8	−2	12.63	56	−6	−2	4.72	50	−10	0	9.44	50	−20	2	6.92
Middle temporal gyrus	21	L	−58	−4	−8	6.10	−60	−54	10	4.66	−66	−30	0	8.25	−56	−46	6	8.65
	21	R	46	−38	2	5.74	54	−50	4	3.77	54	−6	−4	11.10	54	−2	−8	5.46
	37	L									−54	−68	8	4.63	−46	−62	10	6.29
	37	R	50	−60	10	4.52	48	−62	6	5.75	52	−58	10	4.42	46	−62	6	6.00
	39	L	−48	−66	14	3.36	−52	−64	18	3.90	−48	−66	12	5.11	−44	−60	20	3.13
	39	R	38	−70	16	3.37	52	−60	20	3.16	38	−68	12	6.29
Temporal pole	38	L	−42	16	−18	2.88					−46	10	−26	4.55	−56	6	−10	5.23
	38	R	38	16	−20	6.90	52	2	−8	3.23	40	14	−20	7.51	40	10	−26	6.27
Fusiform gyrus	37	L	−40	−44	−14	4.40	−40	−62	−16	4.71	−44	−48	−12	12.18	−40	−50	−10	7.30
	37	R	38	−48	−14	5.04	36	−46	−14	4.04	32	−46	−16	4.52	40	−54	−10	6.09
	19	L	−32	−80	−8	6.34	−34	−70	−14	5.48	−32	−56	−10	5.91	−32	−78	−8	6.04
	19	R	26	−74	−10	3.67	36	−60	−8	5.68	40	−48	−8	6.92	28	−76	−10	5.84
Lingual gyrus	18	L	−26	−78	−6	5.61	−24	−82	−2	4.95	−20	−82	−4	8.85	−22	−80	−8	6.81
	18	R	24	−80	0	7.96	12	−86	0	6.38	22	−78	−2	8.51	16	−80	0	6.63
Middle occipital gyrus	19	L	−32	−88	8	6.97	−46	−84	10	3.91	−32	−86	8	7.61	−46	−84	8	7.08
	19	R	28	−82	4	8.18	34	−82	8	4.68	30	−84	8	10.65	40	−74	4	5.89
Inferior occipital gyrus	17/18	L	−36	−84	−4	10.55	−30	−92	4	6.88	−34	−82	−4	9.51	−28	−86	−6	8.02
	17/18	R	38	−76	−2	7.50	14	−90	−4	4.06	40	−74	0	9.42	20	−88	2	6.80
Cuneus	18	L	−12	−94	20	3.39	−18	−94	12	4.03	−4	−92	12	7.30	−8	−92	10	2.59
	18	R	10	−90	14	4.86	18	−88	10	6.07	16	−88	10	6.73	16	−92	12	6.08
Cerebellum		L	−28	−74	−28	5.34					−26	−72	−32	5.12	−30	−70	−26	3.90
		R	12	−80	−20	5.71					16	−66	−20	4.08

Abbreviations: ASD, autism spectrum disorders; BA, Brodmann area; H, hemisphere; L, left; R, right; TD, typical development.

^*Thresholds were P<.05 corrected for multiple comparisons at the cluster level (k ≥115 voxels) and P<.05 at the voxel level (t>2.57 at peak).

^†Survive only a small volume correction (k ≥55 voxels).

As predicted, when explicitly instructed to attend to the facial expression or tone of voice, children with ASD did show significant activity in regions recruited by TD children when instructions were neutral, specifically in the MPFC and inferior frontal gyrus. Increases in activity in the inferior frontal gyrus were bilateral in the attend to face condition and restricted to the right hemisphere in the attend to prosody condition. Because instructions to attend to facial expression and tone of voice yielded very similar effects overall (both within and between groups), we present the data averaged across the 2 attend conditions compared with rest (Table 4 and Figure 2B).

Figure 2.

Brain activity during potentially ironic scenarios relative to rest. A, Significant activity was observed in the medial prefrontal cortex and the left inferior frontal gyrus in the typically developing (TD) group but not the autism spectrum disorders (ASD) group when instructions were neutral. B, Both groups engaged prefrontal regions when explicit instructions were provided to attend to the facial expression or tone of voice of the speaker. Activation exceeds thresholds of P<.05 corrected for multiple comparisons at the cluster level (k≥115) and P<.05 at the voxel level (t>2.57 at peaks). Yellow arrowheads indicate medial prefrontal cortex; green arrowheads, left inferior frontal gyrus.

Direct comparisons between groups confirmed that TD children did recruit the MPFC and STG more strongly than children with ASD during the neutral instructions condition. Greater cerebellar activity was also observed in the TD group relative to the ASD group. No regions were more strongly activated in children with ASD than in controls. For the attend conditions, TD children engaged the STG, cerebellum, and visual cortices more strongly than children with ASD. However, MPFC activity was no longer significantly different between the groups once specific instructions were provided. A group × condition interaction in the MPFC revealed that activity in this region differed significantly between the groups when specific vs neutral instructions were given. That is, whereas children with ASD showed significantly greater activity in the MPFC when told explicitly to attend to the facial expression or tone of voice than when instructions were neutral, children with TD showed the same amount of activity in this region regardless of instructions (Table 5).

Table 5.

Peaks of Differential Activity During Irony Conditions Given Neutral and Specific Instructions^*

Anatomical Region	BA	H	x	y	z	t Score
Simple main effects, group
Neutral instructions vs rest, TD>ASD
Medial prefrontal cortex	9	R	2	50	20	2.94
	10	L	−6	58	22	2.88
Superior temporal gyrus	42	L	−52	−18	10	4.06†
	22	L	−60	−24	2	4.63†
	22	R	56	−8	−2	6.21†
Middle temporal gyrus	21	R	48	−10	−14	2.63†
Cerebellum		R	10	−78	−22	3.55
Attend face or prosody vs rest, TD>ASD
Superior temporal gyrus	42	L	−52	−20	10	3.25†
	22	L	−60	−22	6	4.46†
	22	R	56	−8	−2	5.47†
Middle temporal gyrus	21	R	48	−12	−8	3.18†
Cuneus	18		0	−90	20	2.97
	18		0	−80	10	2.85
Cerebellum		L	−10	−76	−22	2.62
		R	10	−78	−28	3.37
Group × condition interaction
Attend face or prosody vs neutral instructions, ASD>TD
Medial prefrontal cortex	8	R	10	42	36	3.63†
	9	L	−8	56	26	3.27†
	10	L	−4	58	18	3.67†

Abbreviations: ASD, autism spectrum disorders; BA, Brodmann area: H, hemisphere; L, left; R, right; TD, typical development.

^*P<.05 corrected for multiple comparisons at the cluster level (k≥115 voxels); P<.05 at the voxel level (t>2.57 at peak).

^†Significant after covarying out verbal IQ.

Analyses of covariance confirmed that the group × condition interaction in the MPFC was not due to between-group differences in verbal abilities. Likewise, the significant main effect of group (TD>ASD) in the STG was confirmed both when instructions were neutral and when they were specific to attend to the face or voice (relative to rest). However, after accounting for VIQ, between-group differences were no longer significant in the MPFC or cerebellum during the neutral instructions vs rest comparison or in the cerebellum or visual cortex during the attend to face or prosody vs rest comparison.

Correlations With Social Responsiveness

To relate brain activity to autistic symptoms, we conducted a multiple regression analysis in the ASD group. Controlling for VIQ, a significant negative correlation was found across all irony conditions between activity in the MPFC and social impairment as assessed by the Social Responsiveness Scale.⁴¹ That is, participants with a higher level of social functioning showed greater MPFC activity when attempting to infer a speaker’s communicative intent (Figure 3). Activity in the posterior middle temporal gyrus (Brodmann area 37) was also negatively correlated with the degree of social impairment.

Figure 3.

Activity in the medial prefrontal cortex as a function of symptom severity. A negative correlation was found in children with autism spectrum disorders between activity in the medial prefrontal cortex and scores on the Social Responsiveness Scale (higher scores indicate greater social impairment). Activation exceeds thresholds of P<.05 corrected for multiple comparisons at the cluster level (k≥115) and P<.05 at the voxel level (t>2.57 at peaks).

COMMENT

We found significant differences in brain activity in children and adolescents with ASD compared with TD controls during the interpretation of a speaker’s communicative intent. First, as compared with TD children, children with ASD exhibited a less selective response to potentially ironic scenarios relative to unambiguous scenarios as indicated by reduced activity in the MPFC and right STG. Second, a significant group × condition interaction in the MPFC showed that regional activity was modulated by explicit instructions to attend to facial expression and tone of voice only in children with ASD. However, regardless of instructions, less activity was observed in the STG bilaterally in children with ASD relative to TD children during the irony conditions compared with rest. Finally, MPFC activity was inversely related to symptom severity in the ASD group such that greater social competence was associated with greater activity in the MPFC across all irony conditions. Importantly, these differences were significant after controlling for VIQ.

A less selective neural response to ironic scenarios relative to straightforward scenarios in children with ASD coheres with behavioral evidence that these individuals often mistakenly attribute a literal meaning to ironic utterances.^37,49 Previous research suggests that the MPFC is important for understanding others’ mental states⁵⁰ and that right-hemisphere temporal regions are engaged when coherence seeking is required.^51–54 Selective activity in these regions in TD children for ironic scenarios likely reflects a greater need to integrate facial, prosodic, and contextual cues to infer the speaker’s intent when the literal meaning of a remark conflicts with other available information. Reduced differential activity in this network in children with ASD may indicate less integrative reasoning about intent. Several other studies have also observed abnormalities in MPFC activity in individuals with ASD using a variety of tasks, including theory of mind^5,27–29 and semantic processing.²⁵ The one region where the ASD group showed greater selective activity than the TD group for ironic vs control scenarios was the angular gyrus, known to play an important role in semantic processing. Greater activity in the right angular gyrus in children with ASD than in controls could reflect more reliance on semantic processing at the word or sentence level, perhaps at the expense of integrative processing. However, once VIQ was taken into account, between-group differences in this region were no longer significant. This suggests that increased activity in the ASD group may reflect impaired verbal abilities rather than a true difference in semantic representation.

Perhaps most interesting is the finding that for children with ASD, explicit instructions to attend to facial expression or tone of voice elicited significantly greater activity in the MPFC than neutral instructions simply to pay attention. In contrast, TD children showed significant MPFC activity irrespective of instructions, consistent with evidence that this region is normally recruited automatically when processing communicative intentions.⁵⁵ Our findings extend previous work suggesting that attention to crucial aspects of social stimuli can yield increased activity in regions that typically respond preferentially to such stimuli. Hadjikhani et al⁵⁶ observed comparable levels of FG activity in individuals with ASD and controls when faces were presented with a red fixation cross at approximately eye level. Similarly, Dalton et al¹⁸ found that in individuals with ASD, FG activity increased with time spent looking at the eyes. With respect to understanding others’ intentions, we recently found that children with ASD engaged neurocircuitry similar to that of TD controls when task demands implicitly required attention to prosodic or contextual cues to detect ironic intent.⁵ However, in a natural communicative setting, relevant cues are not experimentally highlighted. Interpreting irony correctly is likely to involve selectively attending to crucial cues (eg, facial expression and tone of voice) in a dynamic environment and integrating this information to reason about mental states.⁵⁷ Here, increased MPFC activity in the ASD group following explicit instructions to attend to facial expression and tone of voice may reflect more reasoning about a speaker’s communicative intent afforded by increased attention to these important cues. The significant association between MPFC activity during irony conditions and social competence in the ASD group suggests that recruitment of this region when inferring communicative intent is indicative of greater success in real-world social situations.

Regardless of task instructions, reduced activity was observed in the STG bilaterally in the ASD group relative to controls. Previous research has demonstrated that the superior temporal sulcus or STG plays a role in polymodal sensory integration and shows a greater response to congruent audiovisual stimuli than to either auditory or visual stimuli alone.^58–60 Reduced STG activity during irony conditions in children with ASD could reflect difficulty detecting the correspondence between facial expression and tone of voice. This notion is consistent with behavioral studies suggesting that children with autism are impaired in matching facial and vocal affect.^61,62

To our knowledge, this study is the first to demonstrate that explicit instructions to attend to important social stimuli can elicit greater activity in the MPFC, a region normally recruited automatically while attempting to infer the communicative intent of another person. Along with other work,^16–19,56 our results suggest that previous reports of hypoactivation in regions supporting the processing of socially relevant information may result from a primary impairment in social interest rather than a fundamental deficit in neural functioning. A lack of attention to faces, voices, and other social stimuli may impair the development of expertise in perceiving and using social cues as well as the automatic engagement of relevant neural circuitry.^63,64

The finding that specific instructions to attend to important social cues resulted in a normalization of MPFC activity in children and adolescents with ASD suggests a strategy for intervention. Adolphs et al⁶⁵ recently showed that explicitly instructing a patient with bilateral amygdala damage to look at the eyes enabled normal recognition of fearful expressions. Similarly, our findings suggest that instructing individuals with ASD to attend to faces and voices may facilitate the extraction of information necessary for interpreting others’ communicative intentions. An approach that combines instruction to use top-down cognitive strategies with perceptual training and reinforcement to facilitate bottom-up attentional processes could capitalize on cognitive strengths while addressing weaknesses in automatic mechanisms. Attentional training in different contexts should result in greater recruitment of the MPFC, perhaps enabling more efficient integration of socially relevant cues. In turn, this may lead to greater success in understanding others’ intentions in everyday interactions. Attention to faces and voices may then become more rewarding and more automatic for individuals with ASD, ultimately leading to higher levels of social responsiveness.

This study has some limitations. First, instructions to attend to the face and voice did not result in enhanced behavioral performance in the ASD group. This is likely due to the simplicity of the scenarios, which had a visual depiction of the event outcome. Performance was excellent even when no attentional guidance was provided. Although prompting the ASD group to attend to facial and prosodic cues may lead to the recruitment of more normative neurocircuitry, behaviorally, more time and practice may be needed before a performance benefit is derived. Compensatory mechanisms developed over time by individuals with ASD are likely to be effective for simple scenarios similar to those used here but are unlikely to be adequate for inferring intent in more dynamic social situations. A second limitation is that the TD and ASD groups differed significantly in VIQ. This leaves open the possibility that our results could be influenced by differences in verbal abilities because task performance increased significantly with VIQ in both groups. However, across all irony conditions, neither VIQ nor accuracy was significantly related to activity in the MPFC, the main site of differences, in either group. A third limitation concerns the possible comorbidity of ASD and ADHD. Although we excluded individuals with a concurrent ADHD diagnosis, some participants were receiving medication, including stimulants. Given that excluding individuals receiving medication did not significantly alter the results and that impairment in theory of mind has not been associated with ADHD,^66,67 we feel it is unlikely that comorbid ADHD significantly impacted our findings. A fourth limitation is that the length of the scenarios in our activation paradigm precluded the use of an event-related design, which would have allowed us to tease apart the neural response to ironic vs sincere scenarios. This compromise was made in the framework of examining the neural processes involved in interpreting communicative intent as a whole rather than assuming that inferring irony and inferring sincerity are independent functional processes.⁶⁸ Finally, the age of our participants varied considerably. Although chronological age was not significantly associated with activity in regions where between-group differences were observed, future research should explore developmental changes in the neural networks supporting the interpretation of communicative intent in ASD.