Perception and deception: Exploring individual responses to deepfakes across different modalities

Abstract

This study is one of the first to investigate the relationship between modalities and individuals' tendencies to believe and share different forms of deepfakes (also deep fakes). Using an online survey experiment conducted in the US, participants were randomly assigned to one of three disinformation conditions: video deepfakes, audio deepfakes, and cheap fakes to test the effect of single modality against multimodality and how it affects individuals’ perceived claim accuracy and sharing intentions. In addition, the impact of cognitive ability on perceived claim accuracy and sharing intentions between conditions are also examined. The results suggest that individuals are likelier to perceive video deepfakes as more accurate than cheap fakes, but not audio deepfakes. Yet, individuals are more likely to share video deepfakes than cheap and audio deepfakes. We also found that individuals with high cognitive ability are less likely to perceive deepfakes as accurate or share them across formats. The findings emphasize that deepfakes are not monolithic, and associated modalities should be considered when studying user engagement with deepfakes.

1. Introduction

Numerous studies confirm that social media increases citizens’ exposure to news both actively and incidentally [1], improving factual political knowledge [2], facilitating political expression [3], and ultimately paving the way for broader civic participation [4]. However, in recent years, growing misinformation on social media has threatened democratic ideals. Misinformation exists in disparate forms ranging from audio clips only, fake news to cheapfakes. However, the deleterious effects of social media on democracy extend beyond the common forms of misinformation. For example, the recent rise of deepfakes and their inimical threat is a harbinger of a possible “information apocalypse”, where citizens may find it hard to discern truth from falsehood [5].

Deepfakes are a relatively new form of misinformation harboring the malicious intent to deceive individuals using cutting-edge artificial intelligence (AI) and nefarious actors to mimic real-life events. This also means that one could be portrayed in a deepfake doing or saying something they did not do without their consent, as exemplified by the many instances of deepfake pornography [6]. Deepfakes threaten organizations as they risk privacy and security [7]. Well-made deepfakes can erode public trust in government [8] and public figures [5] and disturb political stability [9,10]. Exposure to deepfakes can also raise skepticism about news content [11,12] and distrust of news [13].

The elusive dangers of deepfakes have gained much scholarly attention [5,6,8,11,12,14]. However, limited studies directly compare differences in individuals’ vulnerabilities to different forms of deepfakes, including video deepfakes, video cheapfakes, and audio deepfakes. Cheapfakes are a relatively new audio-visual (AV) manipulation involving cheap, accessible software or none at all [15].

Additionally, we are unaware of which modality plays a more significant role in influencing one's perceived claim accuracy or sharing intentions of deepfakes. Hence, to contribute to the existing literature, this study investigates if individuals are more vulnerable and are likely to share video deepfakes against other forms of deepfakes (e.g., audio deepfakes and cheapfakes). Further, given the importance of cognitive ability in misinformation discernment [16,17], we also explore whether cognitive ability influences the vulnerabilities to different forms of deepfakes (i.e., video deepfakes, video cheapfakes, and audio deepfakes).

1.1. Modality, perceived claim accuracy, and sharing intention of deepfakes

Each modality contains unique characteristics, and individuals encode modality-specific content when processing information [18]. Therefore, combining multiple modalities can provide complementary information and more robust inferences, as demonstrated by its vivid applications in AV speech recognition [19], emotion recognition [20], and language and vision tasks [21]. Hence, in a multimodal content like deepfake, many modalities such as facial cues [22], hand gestures [23], body posture [24], and speech cues [25], and tone of the voice [26] can interact to influence one's perceived claim accuracy.

Past research also lends support as human information processing is proven to be limited by attentional resources [27]. This means that humans selectively choose a certain quantity of sensory input to process while the other sensory inputs are neglected. Therefore, the amount of attention paid affects the amount of information absorbed and processed by an individual. Given that an individual has a limited capacity to comprehend informational messages, attempting to process “too much” information may exceed one's limit, failing to comprehend the information [28]. As information processing is critical in the decision-making process, this accentuates the need to compare different modalities as the amount of attention paid to each modality differs and may significantly influence how they process the information.

Even though deepfakes leverage the advantage of boosted believability through their extended span of modalities. Scholars have found that dissonance between the visual characteristics (e.g., loss of lip-syncing, unnatural facial expressions) tends to capture users’ attention and act as an indicator of content veracity [29]. Blatant mismatches are frequent in cheapfakes that cannot achieve seamless audio and visual sync. Research has shown that dissonant AV elements induce a higher user cognitive load and are less likely to perceive the two modalities as a single stimulus, resulting in lower believability [30].

Many studies have explored the realms of video modality and strived to provide a plausible explanation for why video modality is more persuasive and shared. However, more evidence is needed for audio clips. The power of audio lies in its characteristics - speech rate, pitch, volume, and tone, all of which can influence the speaker's perceptions. For example, fast speakers are typically seen as more competent, truthful, and persuasive [31]. Conversely, signs of stress in a speaker's voice may be perceived as less competent and persuasive [32] - which may explain why some users may mistake real audio clips as fake as people may be stressed while speaking in real life. On the other hand, extreme levels of emotion in vocal tone may backfire as individuals tend to perceive that as a lack of realism [33,34].

Nevertheless, we lack empirical evidence for a direct comparison between these multimodal (video deepfakes and cheapfakes) and single-modality misinformation (deepfake audio clips) to determine if having multiple modalities is more influential in users’ decision to believe in the media. Therefore, lending support from past studies on the importance of sensory modalities in the decision-making process, we hypothesize.

H1

Individuals are more likely to perceive the fabricated claims in video deepfakes as accurate than in cheapfakes and audio deepfakes.

Beyond the perceived accuracy of different forms of deepfakes, we also enquire about individual intentions to share the content. Scholars found that individuals are more likely to perceive fake news as credible when presented in a video format than audio and text due to the positive credibility effect [35]. This effect is further strengthened when individuals' issue involvement is low as they will pay less attention and process the information superficially. Likewise, relatively less informed individuals will be more susceptible to believing fake news. As people perceive misinformation as accurate, they tend to believe more in it and become more willing to share it with others online [14,36], accelerating the dissemination of fake news online. The cyclic nature of this spread of online falsehood is extremely minacious to the heavy consumers of online information. While some have concluded that sensory modalities of fake news are noteworthy in influencing one's perceived accuracy and sharing intentions on fake news [35], no studies have paid attention to its role in the sharing intention of deepfakes. This lends credibility to our research direction, and hence, we aim to answer the following research question.

RQ1

Are individuals more likely to share video deepfakes over cheap fakes and audio deepfakes?

1.2. The contingent role of cognitive ability

Cognitive ability is general intelligence and includes the capacity to “reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly, and learn from experience” [37]. Past research has posited that most individuals would not apply their cognitive skills when evaluating information unless needed [38]. The limited capacity model posits that individuals focus on only a few salient features due to limited cognitive space rather than processing all available features in a message when encoding new information [28]. The same applies when people evaluate a message's credibility [39]. Hence, it is essential to consider cognitive ability in our study as it affects the users' evaluation of misinformation on social media and how much they believe or do not believe in it, directly impacting their sharing intentions.

The ‘realism heuristic’ in deepfakes [6] poses a new challenge to social media users, further blurring the line between reality and falsity. Recent research has observed that users follow a ‘seeing-is-believing’ heuristic [40], where most users mistake deepfakes for authentic videos and have overconfidence in their detection abilities [41]. This, when paired with how most online users lack critical thinking and conscious awareness when reading online news as they consume in a state of automaticity [42], magnifies the risk of deepfakes.

However, for a subset of users, seeing is only sometimes believing. Individuals with higher cognitive ability are better at discerning misinformation from real news [16], and those who believe in misinformation rarely doubt its content [17]. Cognitive ability has also been found to be positively associated with effective information processing [43], better decision-making [44], more healthy skepticism toward unfounded beliefs [45], and better capabilities in risk assessment pertinent to their decision of where and whether they should place their trust [46]. In terms of sharing intentions, low-cognitive individuals are more likely to share deepfakes [36,41]. This is sufficient to assume that cognitive ability plays a core part in buffering against manipulative deepfakes. Given that individuals with higher cognitive ability display healthier skepticism towards unreliable content, we propose our second hypothesis.

H2

Higher cognitive ability individuals are less likely to a) believe and b) share video deepfakes, cheapfakes, and audio deepfakes.

2. Method

2.1. Sample and measures

Using a quota sampling strategy, we recruited a sample of US residents through Qualtrics (N = 309). We then matched the sample frame to population parameters focusing on age and gender for greater representativeness of our findings. The study was approved by the Institutional Review Board at Nanyang Technological University (#635). Informed consent was undertaken from all participants, and the study complies with all the regulations and confirmation.

In the first step, the participants answered questions about their demographic characteristics, political motivations, and cognitive ability. Next, they were randomly assigned to one of the three deepfake conditions: a) video deepfake (n = 105), b) audio deepfake (n = 102), or c) cheap fake (n = 102). Then, the participants watched the video or listened to the audio and answered the questions regarding their perceived accuracy of the claims and sharing intentions. The participants were only informed about the nature of the stimulus at the end.

We use a real-world deepfake video of Vladimir Putin (created by RepresentUS) to increase the validity of our findings. The original deepfake was edited to create a cheapfake (video distorted) and audio-deepfake (video layer removed). A screenshot of the video deepfake is included in Fig. 1.

Fig. 1.

A screenshot of the video deepfake.

We also explored if there are characteristic differences between the three experimental conditions. ANOVA and Chi-square tests suggest that the three conditions do not differ across their demographic characteristics, political motivations, or cognitive ability. The results are presented in Appendix A.

The perceived claim accuracy of misinformation was measured by asking respondents to rate their level of agreement (1 = not at all accurate to 5 = extremely accurate) for the claim present in the stimulus “Vladimir Putin mocked Americans for being the reason behind the fall of America?”.

Sharing intention was measured by asking respondents how likely (1 = extremely likely to 5 = not at all) they are to share the video/audio on their social media. The response options were reversed, so a higher value represents greater sharing intentions.

Cognitive ability was measured by the word sum test, where participants were required to match the source word to a closely associated word from a target list of five words. The test includes ten questions. The test shares a high variance with general intelligence and is frequently used to measure the cognitive ability of individuals [47,48]. The correct responses to the ten items were summed to create a scale of cognitive ability (M = 4.99, SD = 2.47, α = 0.74).

We also use demographic characteristics and partisanship as controls. Demographics included age (M = 46.05, SD = 18.24), gender (53% female), education (Median = bachelor's degree), income (Median = $5000 to $6999), and race (73% White). Political motivations include participants' political interest (M = 3.26, SD = 1.21, 1 = not at all interested to 5 = extremely interested) and partisanship (M = 3.58, SD = 2.18; 1 = strong democrat to 7 = strong republican).

3. Results

We compared the differences between conditions for perceived claim accuracy - there was a significant effect of conditions on perceived claim accuracy [F (2, 298) = 3.37, p < .05]. Furthermore, post hoc tests showed that individuals were more likely to perceive the deepfake (M = 3.06, SE = 0.13) to be more accurate than the cheap fake (M = 2.55, SE = 0.13, p < .01) but not the audio-deepfake (M = 2.78, SE = 0.14, p = .14). Please refer to Fig. 2 for the means plot. In addition, we also found the main effect of political interest (F = 14.77, p < .01) and cognitive ability (F = 12.33, p < .01), which are later explored through regression models.

Fig. 2.

The means plot for perceived claim accuracy and sharing intention across conditions.

Next, similar to the test of perceived accuracy, we ran another model testing the differences between the conditions for sharing intention. There was a significant effect of conditions on sharing intention [F (2, 298) = 4.34, p < .05]. Post hoc tests suggest that those who watched the video deepfake (M = 2.51, SE = 0.13) were more likely to share it than audio deepfakes (M = 2.07, SE = 0.13, p < .05) and cheapfakes (M = 2.04, SE = 0.12, p < .001). See Fig. 2 for the means plot. We also observed the main effects of cognitive ability (F = 25.62, p < .001).

We further tested the role of cognitive ability in perceived claim accuracy and sharing intention through regression models (see Table 1). Two models were constructed with perceived claim accuracy and sharing intention as dependent variables. The results presented in Table 1 suggest that those with high cognitive ability are less likely to perceive the claims to be accurate (claim: β = −0.154, p < .001) and share them on social media (sharing: β = −0.243, p < .001). In addition, we also found that perceived claim accuracy is positively associated with sharing intention (β = 0.261, p < .001), suggesting that individuals are more likely to share misinformation across AV formats if they believe it to be true.

Table 1.

Regression examining perceived claim accuracy and sharing intention.

	Perceived Claim Accuracy	Sharing Intention
Predictor variable	β	β
Age	−.087	−.166*
Gender (males = 0)d	.060	−.081
Education	.062	.033
Income	.057	.018
Race (White = 1)d	.015	−.059
Political interest	.162**	.002
Partisanship	.026	−.095
Cognitive ability	−.154***	−.243***
Audio-deepfaked (ref = video deepfake)	−.096	−.120*
Cheap faked (ref = video deepfake)	−.165**	−.115*
Perceived claim accuracy	–	.261***
Total R2	9.5	27.3

Note: *p < .05; **p < .01; ***p < .001; d = dummy.

Finally, we ran two-way ANCOVAs to test the interaction between cognitive ability and different conditions. However, no significant effects were found for either perceived accuracy [F (2, 296 = 1.21, p = .30) or sharing intentions [F (2, 296 = 0.75, p = .48). The means plot is included in Appendix B.

4. Discussion

Our analysis highlights the significance of source and consumer characteristics in user differences in engagement with deepfakes. The findings are discussed in detail below.

First, individuals are more likely to perceive video deepfakes as more accurate than cheapfakes but not audio deepfakes. The results emphasize and reinforce the deceptive notion of deepfakes, highlighting the importance of realism heuristics in AV content. The realism heuristics stemming from the well-made nature of deepfakes may influence the fluency of individuals when they process the information. For example, familiarity elicits a ‘truthiness effect,’ which states that people are more likely to accept information as accurate if perceived as familiar [49,50]. This effect may intensify if the portrayed figure in the deepfake is well-known, such as the one in our deepfake condition - Vladimir Putin. As a stronger sense of familiarity is positively associated with higher fluency, people are more likely to perceive it as accurate regardless of its veracity.

Out of the three deceptive conditions, only the video deepfake and video cheapfakes are multimodal mediums, while the audio deepfake comprises a single modality (audio). Therefore, our findings suggest that the dissonance between AV elements in cheapfakes may be more significant in influencing the perceived claim accuracy than the number and type of modality involved. Past studies have stated the picture superiority effect [51] and that misleading visuals are more duplicitous than illusive verbal content to generate false perceptions based on ‘realism heuristics’ [52].

Subsequently, we expected that people would perceive video deepfakes and cheapfakes more accurately than audio deepfakes; however, we found that people were more likely to perceive video deepfakes and audio deepfakes (single modality) as accurate but not cheapfakes. Our result challenges the popular claim of ocular centrism, the epistemological prioritization of sight above all other senses [53]. It posits that the richness of presented content does not always translate to a more significant role in influencing perceived accuracy. Recent research also confirms that individuals not only fall for video deepfakes [12,14,36] but also cannot reliably recognize audio deepfakes [54].

An explanation can be offered through the uncanny valley theory, which conceptualizes human affinity towards simulated humans regarding their fidelity to reality. As one encounters a simulated human of a certain fidelity, we pick up cues that indicate its belongingness to a human category and cues that indicate belonging to some non-human category [55]. Then, the trend in human affinity for the simulation is expected to follow a curvilinear trend with depression at “average” fidelity. Therefore, a well-made video deepfake and audio deepfake clip may render cues that affirm its humanness; in comparison, the dissonance in AV elements in cheapfake may have cued viewers into its lack of humanness. Hence, individuals have experienced more cognitive dissonance when viewing the video cheapfake than the video deepfake and audio deepfake, resulting in antipathy for the simulated human and triggering the inauthenticity alarm bells within readers to signify to them that it is fake.

Next, regarding sharing intentions, we found that individuals are likelier to share video deepfakes than video cheapfakes and audio deepfakes. There are two possible explanations for this. Firstly, past research has shown the compelling appeal of multimodal information with psychological mechanisms such as source vividness or realism [52]. The Heuristic-Systematic Model (HSM) of information processing posits that individuals rely on simple heuristics, such as the vividness of information when processing via the peripheral route [56] and have the ability to influence behavioral intentions [57]. Second, false news diffuses faster, possibly due to its novelty [58]. Traditionally, novelty attracts attention [59]. Therefore, individuals likely prefer to share video deepfake and cheapfakes because they find them richer in information than audio deepfakes, thus drawing more attention. This canbe supported by the fact that the appeal of rich media also lies in its novelty (Osei-Appiah, 2006).

Nevertheless, comparing the two sets of results, we find that, although people are equally likely to perceive video deepfake and audio deepfake to be more accurate than video cheapfake, this trend is different from the results for sharing intentions. While results from one study are far from establishing broader trends, it may signify the imminent need to consider other factors apart from perceived claim accuracy when studying sharing intentions, as belief is not a necessary precursor to sharing [17]. Hence, our result exemplifies that a deepfake is not monolithic and should be studied as a combination of its characteristics.

Lastly, we found that individuals with higher cognitive ability are less likely to perceive deepfakes as accurate or share them. This aligns with existing literature as those with high cognitive ability are often associated with better decision-making processes [43] due to better elaborative processing and truth discernment [16]. High cognitive individuals spend more time on elaborative processes, and while thinking if a message contains persuasive intent, resistance may occur, and heightened suspicion can negatively influence behavioral intentions [60]. The results confirm that cognitive ability can safeguard against all forms of deepfakes. However, our study focuses on a political deepfake, which may have affected how people assess its quality and accuracy. Hence, this finding may only be generalizable to some genres of deepfakes.

Our findings also apply to real-world scenarios and may be helpful for social media companies and policymakers. While most of the societal attention has been on video deepfakes, the results suggest that the risks associated with audio deepfakes are also significant. This is critical, especially when recent empirical evidence points to the vulnerability of individuals to correctly identify the falsehoods in audio deepfakes [54]. In addition, the findings also highlight the susceptibility of low-cognitive individuals to malicious deepfakes. As such, more focused intervention strategies should be adopted to prevent the citizenry from the threats of deepfakes.

5. Limitations

Finally, we recognize some limitations of the study. First, we chose to use a political deepfake in our study. Hence, the findings may not be generalizable to all disinformation contexts. Second, the study is based on a panel of US citizens participating in online surveys. Therefore, the findings may not generalize to the broader US context or other countries. Nevertheless, given that deepfake exposure and engagement happen primarily within the section of the online population, our findings can inform us of the significant vulnerability of this population. More representative studies are necessary for generalizations.

6. Conclusion

To summarize, this is one of the rare studies to explore the role of sensory modalities in influencing perceived claim accuracy and sharing intentions across different forms of deepfakes, and our results provide empirical evidence that scholars should not use a single lens when studying deepfakes because citizen engagement can differ across forms. The findings have applications for technology policy, where social media companies can focus on sensory modalities for fact-checking and alerting users to potentially manipulated content. In governance, policymakers can draft guidelines to safeguard the more vulnerable section of society (e.g., low cognitive users) against misinformation by including informational cues on social media posts to buffer against misinformation.

Author contribution statement

Saifuddin Ahmed conceived and designed the experiments; performed the experiments; analyzed and interpreted the data; and wrote the paper.

Chua Huiwen analyzed and interpreted the data; and wrote the paper.

Data availability statement

Data will be made available on request.

Supplementary content related to this article has been published online at [URL].

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is/are the supplementary data to this article.