Table 1.
(Types of) Sources used for the analysis of individual deepfake use casesa
| Deepfake disinformation and democracy | ||
|---|---|---|
| Election manipulation | ||
| Source | Type of source b | Description/summary |
| Parkin (2019) | Newspaper article (The Guardian) |
“Politicians fear this like fire. The rise of the deepfake and the threat to democracy” Long-read news article which details uses of deepfake technology for political satire and disinformation. The article also offers an overview over the state of technology and recent advancements, quotes politicians’ and news medias’ fears about future use of deepfakes and discusses potential countermeasures such as detection technologies and public awareness. |
| Goel and Frenkel (2019) | Newspaper article (The New York Times) |
“In India Election, False Posts and Hate Speech Flummox Facebook” Details proliferation of hate speech and disinformation (but not deepfakes) in Indian parliamentary election 2019. |
| Frum (2020) | Article in political magazine (The Atlantic) |
“The Very Real Threat of Trump’s Deepfake” Former US-president Donald Trump retweeted a manipulated image of his political opponent Joe Biden pulling grimaces. The author argues Trump is “testing the boundaries” of posting fakery before the 2020 presidential elections. |
| Cole (2020) | Article in political magazine (Vice) |
“For the Love of God, Not Everything Is a Deepfake” The author argues that the video Donald Trump posted of Joe Biden was not a deepfake and does not severely threaten democracy. |
| Papenfuss (2020) | Newspaper article (Huffpost) |
“Fake Video! Trump Tweets Creepy ‘Independence Day’ Spoof Starring Him” Trump posted a deepfake video of himself starring in “Independence Day”. The article details the video’s content and popular reactions to it. |
| Romm and Stanley-Becker (2020) | Newspaper article (The Washington Post) |
“Suspicious robocall campaign warning people to ‘stay home’ spooks voters nationwide” 10 million voters in Michigan received suspicious calls urging them to stay home and not vote in autumn 2020. The robocalls featured a computerized voice. |
| Collins and Zadrozny (2020) | News media article (NBC News) |
“How a fake persona laid the groundwork for a Hunter Biden conspiracy deluge” A fake “intelligence” document about Joe Biden’s son Hunter Biden was published by a fake persona with a deepfake profile picture. It circulated among right-wing opponents of Joe Biden and was intended to discredit him. |
| Johnson (2020) | Article in online technology magazine (Venture Beat)c |
“Twitter labels video shared by Trump aide as ‘manipulated media’” Details several instances of “cheapfakes” posted by supporters of Donald Trump. |
| Attacking political opponents and suppressing the opposition | ||
| Source | Type of source | Description/summary |
| Satter (2020) | News agency report (Reuters) |
“Deepfake used to attack activist couple shows new disinformation frontier” Several Jewish newspapers published articles by a fake persona with a deepfake profile picture. One article accused two Palestinian activists of sympathizing with terrorists. The article also discusses the targeted newspapers’ differing reactions and the case’s immediate and broader implications. |
| Benzaquen (2020) | Article in political magazine (+ 972 magazine)d |
“‘Leftists for Bibi’? Deepfake pro-Netanyahu propaganda exposed” A right-wing Israeli Facebook group posted several “confessions” by people who allegedly used to support left-wing policies and now support Benjamin Netanyahu. The fake personas used deepfake profile pictures. The article discusses the political context of these “confessions” and their reach. |
| Facebook (2021b) | Company report (Facebook) |
“February 2021 Coordinated Inauthentic Behavior Report” Regular report by Facebook on “coordinated inauthentic behaviour” on its platform, i.e. attempts to manipulate public opinion using fake accounts, and Facebook’s countermeasures. |
| WITNESS (2020a) | Webtalk hosted by human rights organization (WITNESS) |
“Boundary lines? Deepfakes weaponized against journalists and activists: Samantha Cole (Vice) and Nina Schick in conversation with Assia Boundaoui (MIT Co-Creation Studio), Deepfakery Webtalk series, Vol. 4.” Journalists and policy consultants discuss the targeted use of deepfakes as part of a series of webtalks on deepfakes organized by WITNESS, a US-American non-profit organisation specialised in the use of video and other technology to document human rights violations. |
| Jankowicz (2021) | Newspaper article (The Washington Post) |
“Opinion: The threat from deepfakes isn’t hypothetical. Women feel it every day.” Opinion piece by a scholar studying disinformation campaigns stating that deepfakes are being weaponized by foreign actors against women to discourage their political participation and undermine democracy. |
| KrASIA (2021) | Online article by private digital media company (KrAsia)e |
“Did Myanmar’s military deepfake a minister’s corruption confession?” The article reports that a video of a “confession” by a minister in Myanmar stating that he bribed Aung San Suu Kyi might be a deepfake. |
| Foreign interference | ||
| Source | Type of source | Description/summary |
| Bastian (2020) | Article in online technology magazine (Mixed)f |
“Deepfakes. China-Propaganda bei Facebook und Instagram” Reports on Facebook’s deletion of fake profiles on Facebook and Instagram spreading pro-Chinese propaganda. The profiles partially used deepfake profile pictures. |
| Stone (2020a) | Online article by private digital media company (Cyberscoop)g |
“Chinese accounts blast Trump, with help from AI-generated pictures” Reports on the Chinese influence operation “Spamouflage dragon” using fake accounts and criticizing then-US-president Trump. |
| Vavra (2021) | Online article by private digital media company (Cyberscoop) |
“FBI alert warns of Russian, Chinese use of deepfake content” Reports on an alert of the US Federal Bureau of Investigation to US private industry warning of the imminent future use of deepfakes for foreign interference. The alert also details several past foreign influence campaigns using deepfakes. |
| Stone (2020b) | Online article by private digital media company (Cyberscoop) |
“Russia’s IRA used phony news accounts on Facebook to discuss QAnon, coronavirus” Reports on activities of Russian troll farms ahead of the US 2020 presidential elections. The Russian “Internet Research Agency” imitated independent news outlets with fake accounts to spread conspiracy theories and polarize the US public. |
| Graphika (2020) | Company report/analysis (Graphika)h |
“Step into My Parler. Suspected Russian Operation Targeted Far-Right American Users on Platforms Including Gab and Parler, Resembled Recent IRA-Linked Operation that Targeted Progressives” A 36-page in-depth report by analytics firm Graphika analysing alleged Russian disinformation campaigns via a website posing as a right-wing news outlet, via fake Facebook accounts, and on networks such as Gab and Parler. |
| Graphika and DFR Lab (2019: 2–5) | Joint company and think tank report/analysis (Graphika, DFR Lab)i |
“Operation #FFS: Fake Face Swarm. Facebook Takes Down Network Tied to Epoch Media Group That Used Mass AI-Generated Profiles” A 39-page joint report by analytics firm Graphika and the Atlantic Council’s Digital Forensics Research Lab on “Operation Fake Face Swarm”, an anti-Chinese, pro-Trump network of fake profiles on social media using deepfakes. |
| Bastian (2022) | Article in online technology magazine (Mixed) |
“Möglicher Selenskyj-Deepfake: Miserabel und dennoch historisch” Reports on a deepfake video of the Ukrainian president urging Ukrainians to surrender to Russia. The low-quality video had little impact, but better quality deepfakes are within easy reach. The video was also nonetheless “historic” as the first use of deepfakes for foreign political interference on a wider scale. |
| Liar’s dividend | ||
| Source | Type of source | Description/summary |
| Ajder et al. (2019) | Company report/analysis (Deeptrace)j |
“The State of Deepfakes: Landscape, Threats, and Impact” An early, widely cited report by a cybersecurity company on the state of the art, increased commodification, and concrete uses of deepfakes for pornography and in the field of politics. The report is cited here to detail the case of accusations that the New Year’s address by the Gabonese President was a deepfake, which led to riots and a military coup. |
| Blakkarly (2019) | News media article (SBS News) |
“A gay sex tape is threatening to end the political careers of two men in Malaysia” Reports on videos allegedly showing a Malayan minister involved in gay sex acts and the minister’s denial thereof and discusses the case’s implications for LGBT rights. |
| Sudeep (2021) | Newspaper article (Deccan Herald) |
“Is Ramesh Jarkiholi sex video a ‘deepfake’?” Reports on a sex video showing an Indian state minister. The minister insisted the video was deepfaked, and the article discusses the likelihood thereof. |
| Beaumont (2021) | Newspaper article (The Guardian) |
“Donald Trump fans cry betrayal as he rebukes Capitol violence” Donald Trump belatedly condemned the violence committed during the storming of the US Capitol. Conspiracy theorists including QAnon supporters believed the video was a deepfake. |
| MacDonald (2020) | Article on entertainment news website (Inquisitr)k |
“Producers Speculate That Donald Trump’s Post-Coronavirus Video Is A Deepfake” Several producers publicly stated that the video Donald Trump tweeted concerning his coronavirus infection was a deepfake. |
| Chheda (2021) | News media article (International Business Times) |
“Biden’s Head Disappears on Video, Fueling Speculation of Hologram Being Used for His Appearances” Fact-checks (and rejects) accusations that a video of Joe Biden was deepfaked to conceal the president’s allegedly decreasing mental health. |
| Sonnemaker (2021) | News media article (Business Insider) |
“‘Liar's dividend’: The more we learn about deepfakes, the more dangerous they become” Discusses the spread of deepfakes and the associated liar’s dividend, highlighting the case of a Republican congressional candidate denying the veracity of the video showing George Floyd’s murder. |
| Gregory (2021) | Article in online technology magazine (Wired)l |
“Authoritarian Regimes Could Exploit Cries of ‘Deepfake’. Opinion” Opinion piece by Sam Gregory, Program Director at WITNESS on the challenges deepfakes pose for the work of human rights activists. |
| Pfefferkorn (2021) | Think tank analysis (Brookings) |
“The threat posed by deepfakes to marginalized communities” A scholar specialized on cybersecurity and cryptography argues that deepfakes will further undermine marginalized communities’ political standing and, e.g., cast doubt over the veracity of videos documenting police brutality. |
| Weakening news media and journalism | ||
| Source | Type of source | Description/summary |
| Satter (2020) | News agency (Reuters) | See ‘Attacking political opponents and suppressing the opposition’. |
| WITNESS (2020b) | Webtalk hosted by human rights organization (WITNESS) |
“Not funny anymore: Deepfakes, manipulated media, parody and mis/disinformation: Jane Lytvynenko (Buzzfeed News), Karen Hao (MIT Tech Review) and Brandi Collins-Dexter (Color of Change) in conversation with Corin Faife (WITNESS), Deepfakery Webtalk series, Vol. 2.” Journalists and scholars discuss the boundaries between parody and disinformation and what can be done against the latter, as part of a series of webtalks on deepfakes organized by WITNESS. |
| Cohen (2020) | Company report (Jigsaw)m |
“Disinformation is more than fake news. By Jigsaw” Article by a subsidiary of Alphabet Inc. (Google) discussing Assembler, the company’s platform to allow journalists and face-checkers to detect manipulated images. |
| Bundesregierung (2019) | Government report (German federal government) |
“Beschäftigung der Bundesregierung mit Deepfakes. Antwort der Bundesregierung auf die Kleine Anfrage der Abgeordneten Manuel Höferlin, Frank Sitta, Grigorios Aggelidis, weiterer Abgeordneter und der Fraktion der FDP -Drucksache 19/15210” Reply by the German federal government to questions by parliamentarians of the German Liberal party about the government’s response to deepfakes. Details, e.g., the governments understanding of (the problems posed by) deepfakes, the state bodies working on the issue, and existing and required legal regulation. |
| Deepfake hate speech and democracy | ||
| Source | Type of source | Description/summary |
| Schreiner (2019) | Article in online technology magazine (Mixed) |
“Geschichte der Deepfakes. So rasant geht es mit KI-Fakes voran” Continuously updated article which traces the technical development and spread of deepfake images and videos from the invention of generative adversarial networks (GANs) to current technical improvements. |
| Ajder et al. (2019) | Company report/analysis (Deeptrace) | See ‘Liar’s dividend’. The report includes an analysis of websites publishing deepfake porn, targets thereof, and a case study of the app DeepNude. |
| Namboodiri (2021) | Newspaper article (Times of India) |
“Deepfake clips: ‘Sextortionists’ target celebs” Reports on the arrest of three men for blackmailing Indian businessmen, bureaucrats, and celebrities with deepfake sex videos. |
| Hao (2021) | Article in technology magazine (MIT Technology Review) |
“Deepfake porn is ruining women’s lives. Now the law may finally ban it” Details the devastating impact of deepfake pornography on women, its proliferation (including via, e.g., DeepNude), and the inadequacy of victim’s chances to take action against it. The article also reports on current efforts to adapt legal frameworks, e.g. in the USA and UK. |
| Vincent (2020) | Article on online technology portal (The Verge)n |
“Deepfake bots on Telegram make the work of creating fake nudes dangerously easy” Bots on the messaging app Telegram based on the DeepNude code allowed the creation of more than 100,000 fake nude images of women on demand. |
| WITNESS (2020b) | Webtalk hosted by human rights organization (WITNESS) | See ‘Weakening news media and journalism’. |
| Jankowicz (2021) | Newspaper article (The Washington Post) | See ‘Attacking political opponents and suppressing the opposition’. |
aThis table only includes the sources used to detail individual cases or instances of deepfake use, not, e.g., journal articles offering a broader overview of deepfakes’ (potential) impact (unless they also detail individual deepfake use cases). The sources are listed according to their order of citation in the paper. The table was compiled following a suggestion by an anonymous reviewer
bI do not differentiate here between newspaper articles published online and in print. All sources were accessed online
cThe article links both to original sources (tweets and videos), and other news outlets, including CNN, to substantiate the information given
dThe + 972 magazine describes itself as an “independent, online, nonprofit magazine run by a group of Palestinian and Israeli journalists” (see https://www.972mag.com/about/, accessed 27 April 2022). It is, e.g., supported by the German party-affiliated political foundation Heinrich-Böll-Stiftung
eKrAsia is self-reportedly a „digital media company reporting on the most promising technology-driven businesses and trends in the world's emerging markets” (see https://kr-asia.com/about-us, accessed 27 April 2022). The case described here was also taken up by other news media such as Wired (see https://www.wired.com/story/opinion-the-world-needs-deepfake-experts-to-stem-this-chaos/, accessed 27 April 2022)
fMixed describes itself as Germany’s first and most-read online magazine about mixed reality and the future of computing (see https://mixed.de/ueber-mixed/, accessed 27 April 2022). The article links to original sources, including Facebook
gCyberscoop is a “media brand” reporting on cybersecurity (see https://www.cyberscoop.com/contact/, accessed 27 April 2022). The instances of deepfake use reported in this and the subsequent articles on Cyberscoop were also reported on, e.g., by analytics company Graphika, the US Federal Bureau of Investigation, or news media such as the Washington Post. Following respective links from the Cyberscoop articles allowed triangulation to help ensure the validity of the information reported on
hGraphika is a private analytics firm. It analyses the evolution and manipulation of social networks (see https://graphika.com/our-story, accessed 27 April 2022). It has worked together, e.g., with Facebook to uncover and block foreign interference campaigns (see Bastian, 2020)
iThe Atlantic Council is a US-American public policy think tank. Its Digital Forensics Research Lab studies disinformation, exposes human rights abuses and seeks to build digital resilience (see https://www.atlanticcouncil.org/programs/digital-forensic-research-lab/, accesses 27 April 2022)
jDeeptrace is now known as Sensity (AI). The Dutch company works on deep learning technologies to monitor and detect deepfakes (see https://medium.com/sensity/tagged/deepfakes, accessed 27 April 2022). Their 2019 report was widely cited in academia, e.g., by Etienne (2021); Gosse and Burkell (2020); and Ruiter (2021)
kThe website links to the respective tweets, allowing a verification of the information given
lWired is an online magazine reporting on “how technology is changing every aspect of our lives—from culture to business, science to design” (see https://www.wired.com/, accessed 28 April 2022). The article cited here is an opinion piece by a human rights activist frequently cited in the debate on deepfakes and democracy (see e.g., the WITNESS webtalks listed here)
mThe article was published on the online platform Medium
nThe Verge considers itself an “ambitious multimedia effort”. It is a subsidiary of the US company Vox Media (see https://www.theverge.com/about-the-verge, accessed 28 April 2022). The deepfake “ecosystem” this article details was uncovered by Sensity (see footnote j). It was also subject of numerous other media reports (triangulation), e.g., by the BBC (https://www.bbc.com/news/technology-54584127, accessed 28 April 2022) and MIT Technology Review (https://www.technologyreview.com/2020/10/20/1010789/ai-deepfake-bot-undresses-women-and-underage-girls/, accessed 28 April 2022)