Mitigating the harms of manipulated media: Confronting deepfakes and digital deception

Abstract

The ability to distort the visual record is not new. Airbrushed images attempted to alter the historical archives in the early 1900s. Today, digitally manipulated cheapfakes and deepfakes supercharge the spread of lies and conspiracies. While not fundamentally new, today's enhanced ability to easily create, distribute, and amplify manipulated media has heightened the risks. Reasonable and proportional interventions can and should be adopted that would allow for the creative uses of these powerful new technologies while mitigating the risk they pose to individuals, societies, and democracies.

Introduction

My interest in the question of photographic authenticity was first piqued through an otherwise benign encounter. Almost 25 years ago, while standing in line at a library, I casually picked up a book from the return cart. There was no good reason for me to be interested in this book, titled Federal Rules of Evidence (https://www.law.cornell.edu/rules/fre), other than there were no mobile devices back then to be distracted by. Flipping through its pages, I stumbled upon “Article X. Contents of Writing, Recordings, and Photographs, Rule 1001,” which detailed the criteria for admitting an “original” photograph as evidence:

(d) An “original” of a writing or recording means the writing or recording itself or any counterpart intended to have the same effect by the person who executed or issued it. For electronically stored information, “original” means any printout—or other output readable by sight—if it accurately reflects the information. An “original” of a photograph includes the negative or a print from it.

I was surprised to find that the definition of “original” contained such an overly broad and unspecific qualifier “or other output readable by sight.” At the time, the digital revolution was just beginning and digital cameras were still a novelty, photo-editing software was fairly basic, and artificial intelligence (AI) was struggling for relevance. Yet, the direction of technological progress was clear, and it seemed inevitable that digital advancements would significantly complicate the use of photographic evidence in court.

This got me thinking about how a digital image could be authenticated. Two years later—after many false starts and dead ends—I had my first real idea on tackling photo authentication. I was experimenting with Photoshop and spliced a friend's head onto another person's body. As often happens when creating this type of photo composite, the head was too small, so I had to enlarge it. As I did, it struck me that Photoshop had to manufacture the missing pixels to make the head larger. I realized that this type of pixel interpolation should leave behind subtle correlations between neighboring pixels. One of my graduate students at the time and I developed a method to quantify and detect these correlations, and we submitted our findings for publication.

An anonymous reviewer made a perceptive comment that sparked the idea for our second forensic technique. This method leveraged the fact that digital cameras do not capture all the pixels necessary for a full-color image. Instead, they record only a subset and reconstruct the rest. This realization led us to the insight that every original digital image contains a hidden but detectable pattern of color pixel correlations that are disrupted whenever something is added or removed from the image.

Our third forensic technique was inspired by a controversial composite photo depicting presidential candidate Senator John Kerry alongside actress and antiwar activist Jane Fonda. At first glance, the lighting on their faces appeared inconsistent with them sharing the same outdoor stage. Investigating this discrepancy led us to develop a set of forensic methods for analyzing the physical properties of lighting and illumination.

Over the following decade, we developed a suite of photo forensic techniques, each based on characterizing and quantifying irregularities that arise because of different types of manipulation (1, 2). By 2014, I felt that we had a pretty good handle on the problem of photo (and video) authentication. A few years later, however, I started to hear rumblings about a new form of AI-powered manipulation under the moniker of deepfakes. Over the next decade, our nascent and niche field of digital-media forensics rapidly expanded in scope and application, tackling everything from daily press fact-checks to tackling small- to large-scale fraud, and debunking viral online images/videos, global conspiracies, and claims of stolen national elections.

Today, deepfakes (rebranded as generative AI) are both powering new waves of creativity and wreaking havoc on the truth and trust. I describe how deepfakes are created, where they stand in terms of photorealism, how they are being used in new and creative ways, and how they are being used in harmful ways. I conclude with recommendations that I believe are both practical and necessary to protect consumers, organizations, and democracies.

Cheapfakes

While deepfakes refer to AI-powered synthesized or manipulated content, cheapfakes broadly refers to a class of more low-tech manipulated content.

In 2020, 2 videos went viral depicting Speaker Nancy Pelosi appearing intoxicated during public appearances. These low-tech manipulations were created simply by slowing down the original audio to mimic slurred speech (3). No advanced technology was required to convince millions that Pelosi was “blowed [sic] out of her mind,” as one social media post claimed.

Similarly, in June of 2024 (before President Biden withdrew from the presidential campaign) social media was awash in videos edited to play up stereotypes about Biden's age. Media outlets, too, promoted the clips, with the New York Post claiming to have footage showing Biden wandering off in a daze during the G7 summit in Italy. In reality, Biden was congratulating a skydiver who had just landed but was not visible in the frame (4). Context can be dramatically altered by a simple crop—no AI needed.

In some cases, no digital trickery is needed for content to be misleading. Misattribution or staged fakes, consisting of simply mislabeling content, is not uncommon. For example, a viral and heartbreaking photo of a Syrian boy sleeping between the graves of his parents purportedly in war-torn Syria was seen globally as a poignant symbol of the horrific Syrian Civil War. Although the image was not digitally manipulated, it was not from Syria and the boy was not an orphan: the photo was staged by a Saudi photographer (5 ).

Cheapfakes have been with us for quite some time, but now we have to contend with both cheapfakes and their more sophisticated deepfake counterpart.

Deepfakes

Image

A generative adversarial network (GAN) (6) is a common computational technique for synthesizing images of a single category consisting of, for example, people, planes, plants, etc.: generative because these systems are tasked with generating an image; adversarial because these systems pit two separate components (a generator and a discriminator) against each other; and network, because the computational machinery underlying the generator and discriminator are deep neural networks (hence the term deepfake).

StyleGAN (7) is one of the earliest and most effective systems for generating highly realistic human faces (Fig. 2). The generation of a face begins with an initial random arrangement of pixels, which is provided to the discriminator. The discriminator, trained on a vast set of real faces, determines if the generated face is distinguishable from these real faces. If it is, feedback is provided to the generator, which then refines the image and resubmits it for evaluation. This iterative process continues, with the generator and discriminator engaging in an adversarial competition, until the generator produces an image that, to the discriminator, is indistinguishable from real faces.

Fig. 2.

Half of these faces are real and half are AI generated. Can you tell which is which? (The AI-generated faces were generated using StyleGAN2, and the real faces were published in Flickr by their respective authors under either Creative Commons BY 2.0, Creative Commons BY-NC 2.0, Public Domain Mark 1.0, Public Domain CC0 1.0, or US Government Works license.).

While GANs can produce highly realistic images, they offer limited control over the appearance and surroundings of the generated face. In contrast, more recent text-to-image (or diffusion-based) models provide greater flexibility (8). Trained on billions of images paired with descriptive captions, these models gradually corrupt each image with visual noise until only randomness remains. The model then learns to reverse this process, effectively denoising the image conditioned on its descriptive caption. This approach allows for image generation based on text prompts, enabling the creation of scenes such as “A teddy bear smoking and drinking” (Fig. 1).

Fig. 1.

AI-generated images, like this one, created from nothing more than a text prompt (“a teddy bear smoking and drinking whiskey”) can easily and quickly damage a brand or supercharge a conspiracy or disinformation campaign.

Video

Video deepfakes fall into two broad categories: text to video and impersonation.

Text-to-video deepfakes are the natural extension of text to image, in which a model is trained to generate a video to be semantically consistent with a text prompt. A year ago, these systems tasked with creating short video clips from a text prompt like “Will Smith eating spaghetti” yielded videos of which nightmares are made (i.e. https://www.youtube.com/watch?v=XQr4Xklqzw8).

A typical video consists of 24 to 30 still images per second. Generating many realistic still images, however, is not enough to create a coherent video. These earlier systems struggled to create temporally coherent and physically plausible videos in which the interframe motion was convincing. Just a year later, however, these systems have improved tremendously. While not perfect, the resulting videos are stunning in their realism and temporal consistency, and quickly are becoming difficult to distinguish from reality.

Although there are several different incarnations of impersonation deepfakes, two of the most popular are lip-sync and face-swap deepfakes.

Given a source video of a person talking and a new audio track (either AI generated or impersonated), a lip-sync deepfake generates a new video track in which the person's mouth is automatically modified to be consistent with the new audio track. And because it is relatively easy to clone a person's voice from as little as 30 s of their voice (9), lip-sync deepfakes are a common tool used to coopt the identity of celebrities or politicians to push various scams and disinformation campaigns (see The Bad).

A face-swap deepfake is a modified video in which one person's identity, from eyebrows to chin and cheek to cheek, is replaced with another identity. This type of deepfake is most common in the creation of nonconsensual intimate imagery (NCII) (see The Bad). Face-swap deepfakes can also be created in real time, meaning that you will soon not know for sure if the person at the other end of a video call is real or not.

The trend of the past few years has been that all forms of image, video, and audio deepfakes continue their ballistic trajectory in terms of realism, ease of use, and accessibility.

Passing through the uncanny valley

First coined by Japanese roboticist Masahiro Mori in the 1970s, the term “uncanny valley” describes a phenomenon that occurs when a humanoid robot, or an image or video of a computer-generated human, becomes more human-like. There is a point at which the humanoid depiction becomes eerily similar to humans but is still distinguishable from real humans, causing a significant drop in our emotional comfort and acceptance. This transition is known as the uncanny valley. A humanoid depiction is said to exit the uncanny valley when it becomes so realistic that it is indistinguishable from a real person. Generative AI is well on its way to passing through the uncanny valley.

Half of the faces in Figure 2 are real and half are AI generated. Can you tell which is which? (The faces in panels A, B, G, H, and J are real; the faces in panels C, D, E, F, and I are AI generated.) If you are like most others (10), your performance on this task was at near chance.

A recent perceptual study found that when asked to distinguish between a real and AI-generated face like those in Figure 2, participants performed no better than guessing (10). In a second study in which participants were provided with training prior to completing the task, their performance improved only slightly. AI-generated faces are highly realistic and extremely difficult to perceptually distinguish from reality.

Performance is only slightly better for videos of people talking (11). For AI-cloned voices, a recent study found that participants mistook the identity of an AI-generated voice for its real counterpart 80% of the time, and correctly identified a voice as AI-generated only 60% of the time (12).

While not all forms of AI-generated content have passed through the uncanny valley, what remains will almost certainly follow in the near future. We are quickly entering an era in which it is increasingly more difficult for the average person to distinguish between fact and fiction.

The good

Hardly a day goes by when I do not use some form of generative AI in my work, from using large language models (LLMs) to write or debug code to using image synthesis to create visuals for a lecture and summarize the tsunami of daily academic publications. I cannot recall any other technology that has so dramatically and so quickly altered the way I work (and in some cases, think). My colleagues and students report a similar impact in their work and studies.

Beyond personal uses cases, a particularly empowering example of the use of generative AI was by Representative Wexton of Virginia who used an AI-generated version of her voice to address lawmakers on the House floor (13): “My battle with progressive supranuclear palsy, or PSP, has robbed me of my ability to use my full voice and move around in the ways that I used.” Because today's generative AI can clone a person's voice from as little as a 30-s recording, Rep. Wexton was able to speak in her own voice as opposed to the tinny and slightly creepy computer-generated voices of just a few years ago.

On the creative side, generative AI is democratizing access to content creation in a similar way that the Internet democratized access to the publication and consumption of information. What a few years ago may have only been possible with large budgets from major studios can now be done by an single individual and their laptop. We are already seeing, even in these early days of generative AI, new forms of highly creative and inspiring art, music, and film.

I have little doubt that generative AI is offering and will continue to offer positive and exciting use cases, and be an intellectual and creative accelerant, but with a few caveats (14).

Without taking away from these positive applications, I would be remiss if I did not add a few cautionary notes.

Today's LLMs make an A− coder an A+ coder, but it also makes a B+ coder a C− coder. As a computer scientist and experienced coder, when I ask an LLM for a code snippet, I have a fairly good idea of what the right answer will look like, and I understand the proper levels of abstraction needed to build reliable systems. Without this knowledge, however, today's LLMs can easily lead a less experienced programmer astray (15). I am fairly confident that the same can be said about other skill sets beyond coding.

All forms of generative AI have been trained on decades of user-generated content, in many cases without permission and in many cases in direct violation of copyright laws (16). Trying to justify their indiscriminate scraping of online content, OpenAI—one of the leaders in the generative-AI space—admitted that it would be “impossible to train today's leading AI models without using copyrighted materials” (https://www.theguardian.com/technology/2024/jan/08/ai-tools-chatgpt-copyrighted-material-openai). This is a bit of a hard pill to swallow for a company that in less than 10 years has grown to a valuation of over $150 billion. Moving forward, we will have to find a more equitable way for content creators to be compensated as we use the tools trained on the fruits of their labor.

The bad

Nonconsensual intimate imagery

Before the less objectionable term “generative AI” took root, AI-generated content was referred to as “deepfakes,” a term derived from the moniker of a Reddit user who in 2017 used this nascent technology to create NCII (often referred to by the misnomer “revenge porn,” suggesting somehow that the women depicted inflicted a harm deserving of revenge). Seemingly unable to shake off its roots, generative AI continues to be widely used to insert a person's likeness (primarily women and also children) into sexually explicit material that is then publicly shared by its creators as a form of humiliation or extortion.

While it used to take thousands of images of a person to digitally insert them into NCII, today only a single image is needed. This means that the threat of NCII has moved from the likes of Scarlett Johansson, with a large digital footprint, to anyone with a single photo of themselves online.

Shown in Figure 3, for example, is an image that I generated using a free service (that does not allow the generation of explicit material) in which I inserted my face into an image of an inmate in an orange jumpsuit.

Fig. 3.

A deepfake in which I inserted my face (source in upper left) into an AI-generated image of an inmate in an orange jumpsuit.

A recent study surveyed 16,000 respondents in 10 countries and found that 2.2% of respondents reported being a victim of NCII, and 1.8% reported creating NCII (17). Given that many people may not know that they are victims and many may be unwilling to admit creating NCII, this is surely a lower bound. The threats of NCII are neither hypothetical nor relegated to the dark recesses of the internet. Finding NCII content and creation tools is no further away than a Google search.

Child sexual abuse imagery

The Cyber Tipline at the U.S.-based National Center for Missing and Exploited Children (NCMEC) is a national reporting system for reporting all forms of child sexual exploitation including apparent child sexual abuse material (CSAM). The majority of reports come from electronic service providers including the largest social media platforms like Facebook and Instagram. In 2010, NCMEC received 132,000 reports. By 2015, the number of reports grew to over 4 million, and then 21 million in 2020 and 36 million in 2023. The average age of a child depicted in this content is 12 and sometimes is as young as just a few months.

Starting in 2023, NCEMC has received a small but steadily increasing number of reports that appears to be AI generated or AI manipulated. Given the escalating volume of CSAM reports over the past two decades it was, sadly, predictable that this nascent technology would quickly be weaponized in this horrific way (18).

18 U.S. Code § 2252A uses a standard prohibiting any visual depiction of CSAM that is “virtually indistinguishable” from a minor engaging in sexual conduct (19). That is, creation or possession of CSAM can extend beyond material depicting an actual child to material that is computer generated. Beyond the legal standing of AI-generated CSAM, the large-scale creation of abusive content holds the potential to both normalize the sexual abuse of children for offenders and overwhelm an already strained CyberTipline.

While some generative-AI systems placed reasonable guardrails to prevent the creation of CSAM, others did not. Stability AI's first version of their image generator—Stable Diffusion—was open sourced with no guardrails. In response to concerns of potential abuse, the company's founder, Emad Mostaque, said, “Ultimately, it's peoples' [sic] responsibility as to whether they are ethical, moral and legal in how they operate this technology” (https://www.theverge.com/2022/10/18/23410435/stability-ai-stable-diffusion-ai-art-generator-funding-round-billion-valuation). Depending on your viewpoint, this is spectacularly naive, cynical, or simply indifferent.

Fraud

First it was Instagram ads of Tom Hanks promoting dental plans. Then it was TV personality Gayle King hawking a sketchy weight-loss plan. Next, Elon Musk was shilling for the latest crypto scam, and Taylor Swift was announcing a giveaway of Le Creuset cookware. More recently, it has been Brad Pitt and Cristiano Ronaldo promoting phony medicines to treat serious diseases like cancer. All, of course, were deepfake scams.

AI-powered scams are not just impacting individuals, they are also impacting small- to large-scale organizations. Earlier this year, a finance worker in Hong Kong was tricked into paying out $25 million to fraudsters using deepfake technology to pose as the company's chief financial officer in a video conference call.

This was not the first such example. In 2019, a United Kingdom based company experienced the same fate when an imposter used an AI-synthesized voice to steal $243,000 in a similar type of scam. And, in early 2020, a United Arab Emirates bank was swindled out of $35 million when the bank teller was convinced to transfer the funds after receiving a phone call from the purported director of a company whom the bank manager knew and with whom he had previously done business. It was later revealed that the voice was that of an AI-synthesized voice made to sound like the director. These incidents are almost certainly the canaries in the coal mine.

Similar types of fraud are also being carried out at the individual level. In early 2023, the mother of a teenager received a phone call from what sounded like her distressed daughter claiming that the teenager had been kidnapped and feared for her life. The scammer demanded $50,000 to spare the child's life. After calling her husband in a panic, she learned that the daughter was safe at home.

Generative AI is a powerful new weapon in the arsenal of cyber criminals. As synthesized audio and video continue to improve in quality and accessibility, it is reasonable to predict that these technologies will continue to be used to commit a range of small- to large-scale frauds.

Disinformation

By mid-May 2020, during the height of the global pandemic, 28% of Americans believed that Bill Gates intended to use COVID-19 to enforce a mandatory vaccine program with tracking microchips (20). This conspiracy theory was not confined to the United States. Global surveys conducted across Central and South America, the Middle East, Northern Africa, and Western Europe found that 20% of the public subscribed to this unfounded claim (21).

As of this year, 22% of Americans do not believe in climate change, with only 54% believing that climate change is human driven. Understanding of climate change is highly partisan, with 93% of Democrats and only 62% of Republicans believing in climate change (22).

The widespread, far-right QAnon conspiracy claims that a secret cabal of Satan-worshipping, cannibalistic pedophiles was operating a global child sex-trafficking ring and conspiring against Donald Trump. A recent poll found that a 37% of Americans were unsure whether QAnon was true or false, while 17% believed it to be true (23).

Our global health, our planet's health, and our democratic institutions are all under attack due to rampant disinformation, conspiracies, and lies. It seems likely that deepfakes will be an accelerant to disinformation campaigns that until today managed to take significant hold without accompanying visual “evidence.”

Liar's dividend

While the harms from deepfakes are real and already with us, perhaps the most pernicious result of deepfakes and general digital trickery is that when we enter a world in which anything we see or hear can be fake, then nothing has to be real. In the era of deepfakes, a liar is equipped with a double-fisted weapon of both spreading lies and using the specter of deepfakes to cast doubt on the veracity of any inconvenient truths—the so-called liar's dividend (24).

In 2016, for example, Musk was recorded saying “a Model S and Model X at this point can drive autonomously with greater safety than a person. Right now.” After a young man died when his self-driving Tesla crashed, his family sued claiming that Musk holds some responsibility because of his claims of safety. In attempting to counter this claim, Musk's attorneys told the court that Musk “like many public figures, is the subject of many “deepfake” videos and audio recordings that purport to show him saying and doing things he never actually said or did.” Fortunately, the judge was not persuaded, “Their position is that because Mr. Musk is famous and might be more of a target for deep fakes, his public statements are immune,” wrote Judge Evette Pennypacker. She added, “In other words, Mr. Musk, and others in his position, can simply say whatever they like in the public domain, then hide behind the potential for their recorded statements being a deep fake to avoid taking ownership of what they did actually say and do. The Court is unwilling to set such a precedent by condoning Tesla's approach here.”

More recently, in the lead up to the highly contentious 2024 national election, Donald Trump publicly accused the Harris-Walz campaign of posting AI-generated images of large rally crowds (Fig. 4). This claim is baseless. It could be argued that denying crowd size was simply petty, but there could also have been something more nefarious at play. As Donald Trump publicly stated that he would deny the results of the election if he lost, denying large crowd sizes would give him ammunition to claim voter fraud after the election. As the violent insurrection from the previous election showed us, the stakes are quite high.

Fig. 4.

An authentic photo of a Harris-Walz rally that Donald Trump claimed, in a Truth Social post (https://truthsocial.com/@realDonaldTrump/posts/112944255426268462), was fake. Original photo reprinted with permission from Harris Campaign.

As deepfakes continue to improve in realism and sophistication it will become increasingly easier to wield the liar's dividend.

Mitigations

Generative AI continues its ballistic trajectory in terms of its ability to create content that is—or soon will be—nearly indistinguishable from reality (see Passing Through the Uncanny Valley). While there are many exciting and creative applications (see The Good), this technology is also being weaponized against individuals, societies, and democracies (see The Bad).

If we have learned anything from the past two decades of the technology revolution and the disastrous outcomes in terms of invasions of privacy and toxic social media, it is that things will not end well if we ignore, or downplay as the cost of innovation, the malicious uses of generative AI.

I contend that reasonable and proportional interventions from creation through distribution, and across academia, government, and the private sector are both necessary and in the long-term interests of everyone. I will enumerate a range of interventions that are both practical and when deployed properly can keep us safe and allow for innovation to flourish.

Creation

When text-to-image image generators (see Deepfakes) first splashed onto the scene, Google initially declined to release its technology while OpenAI took a more open, and yet still cautious, approach, initially releasing its technology to only a few thousand users. They also placed guardrails on allowable text prompts, including no nudity, hate, violence, or identifiable persons. Over time, OpenAI has expanded access, lowered some guardrails, and added more features. Stability AI took yet a different approach, opting for a full release of their Stable Diffusion with no guardrails. And most recently, Elon Musk's image generator, Grok, followed a similar course, leading to all sorts of ridiculous content, from Kamala Harris romantically embracing Donald Trump to Mickey Mouse wielding an AR-15, to the more offensive and dangerous.

Regardless of what you think of Google's or OpenAI's approach, Stability AI and Grok made their decisions largely irrelevant: when it comes to this type of shared technology, society is at the mercy of the lowest common denominator. Nevertheless, generative-AI systems should follow several simple rules to mitigate the harm that comes from their services, and the remaining bad actors will have to be dealt with through legislation and litigation (see the following).

1. The Coalition for Content Provenance and Authentication (C2PA) (https://c2pa.org) is a multistakeholder, open-source initiative aimed at establishing trust in digital audio, image, and video. The focus of the C2PA is creating standards to ensure the authenticity and provenance of digital content. This standard includes the addition of metadata and embedding an imperceptible watermark into content, and extracting a distinct digital signature from content that can identify content even if the attached content credentials are stripped out. Any generative-AI service should implement this standard to make it easier to identify content.

2. Because text-to-image and text-to-video systems are capable of producing content limited only by the imagination of the creator (see Deepfakes), some reasonable semantic guardrails should be implemented on both the input and output. On the input side, a LLM can flag prompts that includes requests for NCII, CSAM, or other violative or illegal content. On the output side, a multimodal LLM can similarly flag violative content that managed to slip through the input guardrails.

3. Although content credentials and semantic guardrails are important steps in mitigating harms, they are not infallible. Generative-AI services should adopt a know-your-customer approach common in all financial institutions. This will both put creators on notice that their content creation is not anonymous, and allow platforms to aid investigations into illegal uses of their services.

Distribution

There are 3 main phases in the life cycle of online content: creation, distribution, and consumption. I have addressed creation in the previous section and address consumption next. On the distribution side, social media needs to take more responsibility for everything from the unlawful to the lawful-but-awful content that is both shared on their platforms and amplified by their own recommendation algorithms (25).

While it is easy to single out social media platforms for their failure to rein in the worst abuses on their platforms, from CSAM, to NCII, fraud, violence, and dangerous disinformation campaigns, these platforms are not uniquely culpable. Social media operates within a larger online ecosystem powered by advertisers, financial services, and hosting/network services.

Each of these—often hidden—institutions must also take responsibility for how their services are enabling a plethora of online harms.

1. In addition to improving on their content moderation policies and enforcement, social media can create a global shared database of identified NCII as they have previously done for CSAM and terror-related content (26). Once NCII is identified, such a shared database would prevent NCII from being reuploaded, thus reducing the continued harm to victims.

2. Pressure to effect change on platforms rarely comes from users because we are not the customer, we are the product. The real customers are advertisers who should wield their power to effect change by insisting, for example, that their products and services not be advertised alongside CSAM, NCII, and violent content. This is not just the right thing to do, it is the smart thing to do for brand protection.

3. The largest financial services (Visa, MasterCard, PayPal, etc.) should not be in business with services that primarily host or produce NCII or other illegal and harmful content. There are at least 2 examples in which financial services were able to effect change when they withheld service from PornHub (for hosting CSAM and NCII) and Backpage (for enabling sex trafficking).

4. While more fraught, computing infrastructure services including GitHub to Amazon/Google/Microsoft cloud and network services like Cloudflare can also act as better stewards. For example, at a hate-filled neo-Nazi march in Charlottesville, Virginia in 2017, violence erupted between marchers and counter-protesters leading to the horrific murder of a counter-protester. In the aftermath, companies like Cloudflare came under heavy criticism for providing services to neo-Nazi groups like Daily Stormer, and for giving them personal information on people who complain about their content (27). Despite initially refusing to act, Cloudflare eventually terminated the account of Daily Stormer (28). While these groups will eventually find another home, that does not mean that we should not continually make the Internet—where they can amplify their hate and violence—an increasingly unwelcome place.

Consumption

When discussing deepfakes, the most common question I’m asked is, “How can the average consumer distinguish the real from the fake?” My answer is always the same: “nothing.” Afterward, I explain that artifacts in today's deepfakes—7 fingers, incoherent text, mismatched earrings, etc.—will be gone tomorrow, and my instructions will have provided the consumer with a false sense of security. The space of generative AI is moving too fast and the forensic examination of an image is too complex (see next section) to empower the average consumer to be an armchair forensic detective.

There are, however, things that consumers can do to protect themselves from the being defrauded or fooled by deepfakes.

1. Protecting against fraudulent phone calls from scammers claiming to be a family member can be as simple as having an agreed upon family code word that would have to be produced when an unexpected or emergency call is received.

2. Protecting against disinformation is, of course, more challenging as more and more people get the majority of their news from increasingly louder echo chambers (29). Here, I propose the development of a national K-12 effort to educate students on how to strike a balance between skepticism and vigilance, how to spot signs of disinformation, how to fact-check, and how to generally be better digital citizens than the previous generation.

3. Protecting against being a victim of NCII effectively requires being completely invisible online, which in today's world is nearly impossible. If you are a victim of NCII, several organizations may be able to provide assistance or advice, including the Cyber Civil Rights Initiative (https://cybercivilrights.org).

Authentication

Identifying manipulated content (image, audio, video) can be partitioned into 2broad categories: (i) active and (ii) reactive. Active approaches include the type of C2PA content credentials described previously (see Creation), while reactive techniques operate in the absence of credentials inserted at the point of recording or synthesis.

Within the reactive category, there are 2 basic approaches to detection: (i) learning based, in which features that distinguish real from synthetic or manipulated content are explicitly learned by any of a range of different machine-learning techniques; and (ii) artifact based, in which a range of low-level (pixel-based) to high-level (semantic-based) features are explicitly designed to distinguish between real and synthetic or manipulated content (30).

An example of a learning-based technique was developed and deployed to detect potentially fraudulent profiles on LinkedIn (31). A standard deep neural architecture (32) with a modest 7.8 million parameters was trained to distinguish real from AI-generated faces (both GAN and diffusion; see Deepfakes). The model takes as input a color image and generates a numerical score in the range [0, 1]. Scores near 0 indicate that the image is likely real, and scores near 1 indicate that the image is likely AI generated.

With a fixed false positive rate of 0.5% (incorrectly classifying a real face), AI-generated faces are correctly classified in training/evaluation at a rate of 100%/98%. Across different synthesis engines (StyleGAN 1, 2, 3; Stable Diffusion 1, 2; and DALL-E 2) used for training, the true positive rate (correctly classifying an AI-generated image) varies from a low of 93.3% to a high of 99.5%. For faces generated by synthesis engines not used in training, the true positive rate drops to 84.5% at the same false positive rate, showing good but not perfect out-of-domain generalization. This type of out-of-domain generalization is a typical weakness of learning-based approaches.

An example of a low-level, artifact-based technique leverages periodic patterns that emerge due to the processing pipeline underlying synthetic image generation (33, 34). Some of these patterns are caused by a common upsampling operation that occurs during synthesis when an initial low-resolution image capturing the overall structure is iteratively increased in resolution to fill in details. The benefit of this approach is that unlike the pure learning-based approaches, the resulting artifacts are more explainable and interpretable.

A simple and elegant example of a high-level, artifact-based technique leverages the same linear perspective cues understood by Renaissance painters as early as the 15th century.

In an authentic image, all parallel lines on a planar surface will converge to a vanishing point. Any gross deviations from these geometric constraints may be an indication of image synthesis or manipulation (35). Shown in Figure 5, for example, is an AI-generated image in which it is reasonable to assume that the lines on the tiled floor are parallel in the 3D scene. In an authentic photo, the 5 annotated lines should converge to a single vanishing point. The lack of a consistent intersection is evidence of synthesis or manipulation. This type of geometric inconsistency is common in today's AI-generated imagery because the underlying synthesis is strictly statistical in nature without an explicit model of camera optics or 3D geometry.

Fig. 5.

An AI-generated image in which I have overlaid 6 constraints (yellow lines) corresponding to parallel lines on a flat surface in the 3D scene. The lack of a coherent intersection (vanishing point) is evidence of synthesis or manipulation.

Collectively, reactive techniques can be effective at exposing manipulated content. There are, however, some fundamental limitations. First, by the time malicious content is uploaded online, flagged as suspicious, and analyzed for authenticity, and a fact-check is posted, the content can easily have racked up millions of views. Second, there is a fundamental asymmetry between detecting manipulated content and verifying the authenticity of real content. In particular, when evidence of manipulation is found (e.g. inconsistent vanishing points), it is possible to say something definitive. When evidence of manipulation is not found, however, it is not necessarily possible to definitively say the content is real—that is, the lack of evidence is not itself evidence.

A deep arsenal of authentication techniques is, nevertheless, important to mitigate the harms from manipulated content [see (1, 36) for a more complete coverage of classic forensic techniques for detecting cheapfakes and (30, 37) for a more complete coverage of detecting deepfakes]. To this end, the nascent field of digital media forensics would benefit from support from other communities.

1. Those on the generative-AI side should make accessible to the digital media forensic research community early access to their next generation synthesis tools along with access to large datasets for development and evaluation of forensic techniques.

2. With billions being poured into generative AI from the private sector, governments should commit to supporting the digital media forensic research community. This can take the shape of programs like DARPA's MediFor and SemaFor, providing access to computing resources that are often out of reach of most academics, and supporting the creation of common evaluation benchmarks.

Legislation

Existing legislation should be sufficient to combat CSAM and fraud, whether AI powered or not. Here, interventions to protect the public and prosecute perpetrators are primarily limited by law enforcement resources and the inaction of the largest social media platforms. With tens of millions of CSAM reports each year to NCMEC, for example, law enforcement is simply overwhelmed. With billions of uploads each day to social media, these platforms are incapable—and too often unwilling—to combat illegal activity on their services.

Most agree that bans or restrictions should be placed on the creation and distribution of NCII, but the law has not fully caught up with the latest technology that now makes it too easy to create and distribute this type of content.

In recent years, however, there has been a patchwork of national and international legislation enacted (38, 39). In 2019, the US state of Virginia expanded its 2014 “revenge porn” laws to include synthesized or manipulated content, making it illegal to share nude photos or videos of anyone—real or fake—without their permission. California, Hawaii, New York, and Texas have similar restrictions. In 2021, Australia amended its laws to include synthesized or manipulated content; violations can incur both criminal charges and monetary fines, and in 2025 the United States passed the Take it Down Act. This Act makes it a federal crime to knowingly publish or threaten to publish NCII, whether authentic or digitally fabricated. And, online platforms are required to remove reported NCII within 48 hours of a valid request from a victim, and they must also take reasonable steps to prevent the reappearance of such content.

Because the Internet is borderless, nations should now band together to move from a patchwork of legislation to a consistent set of rules and regulations to combat NCII. It remains unclear, however, whether legislation can fully rein in these abuses. Hollywood actress Scarlett Johansson—a frequent target of NCII—told The Washington Post, “I think it's a useless pursuit, legally, mostly because the internet is a vast wormhole of darkness that eats itself” (https://www.washingtonpost.com/technology/2018/12/31/scarlett-johansson-fake-ai-generated-sex-videos-nothing-can-stopsomeone-cutting-pasting-my-image/).

With some exceptions including speech designed to interfere with elections or the peaceful transfer of power, mitigating the harms from various forms of political speech is complex, and legally fraught. It is not, after all, illegal for a politician to lie or for anyone to believe those lies.

Nevertheless, several states have recently passed legislation designed to protect the integrity of elections from misleading deepfakes. In 2024, in the lead-up to a contentious national election in which deepfakes had already played a roll, both the states of Minnesota and California passed legislation to impose varying civil and criminal penalties to those creating, distributing, and in some cases, hosting AI-powered election misinformation. These laws are not without controversy and they will soon be challenged on First Amendment grounds.

Nevertheless, practical and proportional responses to existing and emerging threats are within reach.

1. Despite Scarlett Johansson's perfectly reasonable assessment of the state of the Internet, a combination of updating of existing legislation and crafting new legislation to combat emerging threats is necessary, if not sufficient. To date, only a handful of nations have moved to mitigate the harms from deepfakes. While I applaud these efforts, Internet regulation cannot be effective with a patchwork of laws. A coordinated international effort is required. In this regard, the European Union's Digital Safety Act, the United Kingdom's Online Safety Act, and Australia's Online Safety Act provide a road map for others. While regulation at a global scale will not be easy, some common ground can surely be found among the United States and its allies, thus serving as a template for other nations to customize and adopt.

2. In the absence of sweeping legislation, liability can be a powerful motivating factor for the technology sector to make sure that their products and services are not harmful. But, penetrating the powerful liability shield of Section 230 of the Communications Decency Act has proven challenging. Written in 1996, Section 230 provides broad immunity to online platforms (including social media) from being held liable for user-generated content, and it allows platforms to moderate content in “good faith” without being treated as the publisher or speaker of the content. The US Congress has repeatedly tried (and failed) to modernize this outdated law that could not and does not work in today's modern technology landscape. The US Congress needs to revisit the issue by modernizing Section 230 to create some liability to motivate a mindset of safety by design, not safety as (at best) an afterthought.

3. On the specific issues of CSAM and NCII, more resources and training should be provided to law enforcement to provide resources for victims and support for investigations and, where appropriate, prosecutions.

Academe

In criticizing the reckless use of scientific advancements without considering the ethical implications, Jeff Goldblum's character, Dr. Ian Malcolm, in the 1993 blockbuster movie Jurassic Park, said, “Your scientists were so preoccupied with whether they could, they didn’t stop to think if they should.”

I am, of course, not equating advances in AI with the fictional resurrection of dinosaurs some 66 million years after extinction. The spirit of Goldblum's sentiment, however, is one all scientists should absorb.

Many of today's generative-AI systems used to create NCII and CSAM (see The Bad) are derived from academic research. For example, pix2pix developed by University of California, Berkeley, researchers uses a GAN (see Deepfakes) to transform the appearance or features of an image (e.g. transforming a daytime scene into a nighttime scene) (40). Shortly after its release, this open-source software was used to create DeepNude, a software that transforms an image of a clothed woman into an image of her unclothed (41). The creators of pix2pix could and should have foreseen this weaponization of their technology and developed and deployed their software with more care.

This was not the first such abuse, nor will it be the last. From inception to creation and deployment, researchers need to give more thought on how to develop technologies safely and, in some cases, if the technology should be created in the first place.

1. During the peer-review process, reviewers should assess if any ethical or safety concerns should be considered and/or addressed by the authors prior to publication.

2. While open-source deployments are of great benefit to the larger research community, this benefit should be counterbalanced by the potential risks (as we saw in the previous example).

3. At both the undergraduate and graduate levels, mandatory curricular additions are needed to expose math and engineering students to more ethics, history, philosophy, political science, and a broader swath of the liberal arts than most typically see. Our future innovators need the proper scaffolding to think about the broader issues of how technology is intersecting with society and the world beyond Silicon Valley.

Conclusions

There is much to be excited about in this latest wave of the technology revolution. But, if the past few technology waves have taught us anything, it is that left unchecked, technology will begin to work against us and not for or with us. We need not make the mistakes of the past. We are nearing a fork in the road for the type of future we want and what role technology will play.

Famed actor and filmmaker Jordan Peele's 2018 public service announcement on the dangers of fake news and the then nascent field of deepfakes (https://www.youtube.com/watch?v=cQ54GDm1eL0) offers words of advice and caution. The public service announcement concludes with a Peele-controlled President Obama saying, “How we move forward in the age of information is gonna be the difference between whether we survive or whether we become some kind of f***ed up dystopia.” I could not agree more.

Funding

This work was funded by a grant from the University of California Noyce Initiative.

Data Availability

There are no data underlying this work.