Advances in artificial intelligence: a review for the creative industries

Abstract

Artificial intelligence (AI) has undergone transformative advances since 2022, particularly through generative AI, large language models (LLMs), and diffusion models, fundamentally reshaping the creative industries. However, existing reviews have not comprehensively addressed these recent breakthroughs and their integrated impact across the creative production pipeline. This paper addresses this gap by providing a systematic review of AI technologies that have emerged or matured since our 2022 review, examining their applications across content creation, information analysis, post-production enhancement, compression, and quality assessment. We document how transformers, LLMs, diffusion models, and implicit neural representations have established new capabilities in text-to-image/video generation, real-time 3D reconstruction, and unified multi-task frameworks–shifting AI from support tool to core creative technology. Beyond technological advances, we analyze the trend toward unified AI frameworks that integrate multiple creative tasks, replacing task-specific solutions. We critically examine the evolving role of human-AI collaboration, where human oversight remains essential for creative direction and mitigating AI hallucinations. Finally, we identify emerging challenges including copyright concerns, bias mitigation, computational demands, and the need for robust regulatory frameworks. This review provides researchers and practitioners with a comprehensive understanding of current AI capabilities, limitations, and future trajectories in creative applications.

Introduction

The influence of artificial intelligence (AI) has grown dramatically over the past few years, particularly due to the rise of generative AI and large language models (LLMs). These advancements are widely regarded as beneficial by many countries, creating significant opportunities for growth (e.g. as outlined in the UK, by the Authority of the House of Lords (Communications and Digital Committee 2024)). These advances have also had significant direct and indirect impacts on the creative industries, influencing the direction of their growth. Generative AI, for instance, primarily focuses on generating new data that is not identical to the training data yet shares similarities with it. However, the cardinality of the training data can be huge, larger than what any individual human has ever encountered. The resulting output may therefore act as a new source of inspiration.

AI tools also provide opportunities for a wider range of users to work more efficiently and effectively, with even greater creativity. Moreover, these new technologies not only influence creators, but they also enable new ways for audiences to experience art and culture (Jeary and Gajjar 2024).

A major breakthrough in generative AI has been led by OpenAI,1 an AI research and deployment company, with their introduction of Generative Pre-trained Transformer (GPT) models for LLMs. LLMs are specifically designed to understand and generate human language. They are characterized by their vast size in terms of parameters and the amount of training data used to create them. This breakthrough was particularly impactful when the company released ChatGPT in 2022, which was fine-tuned from a model in the GPT3.5 series. ChatGPT is a conversational model that includes advanced safety features that mitigate the generation of inappropriate content. Several other LLM platforms were also developed contemporaneously, such as LaMDA and PaLM by Google AI, Ernie Bot by Baidu, and BLOOM by BigScience. Additionally, Anthropic launched Claude, the LLM trained specifically to be harmless and honest, leveraging reinforcement learning from human feedback (RLHF) - a technique used to train AI systems to appear more human (Bai et al. 2021). Nonetheless, ChatGPT stands out as the most renowned, thanks to its quick and efficient responses, and notably its public accessibility, being available for free.

Another breakthrough in 2022 was in the area of text-to-image models. OpenAI achieved a significant milestone with DALL E 2, producing impressive artworks and photorealistic images despite its limited language understanding. Midjourney by Midjourney, Inc., another well-known text-to-image generator, supports image resolutions of 10241024 pixels and also provides upscaling tools. Stable Diffusion by Stability AI, for which the code and model weights are publicly available,2 allows developers and artists to further adapt AI to suit their own specific applications.

The next breakthrough occurred in 2023 when OpenAI unveiled GPT-4, a significantly larger model with third-party estimates of 1.8 trillion parameters; OpenAI has not disclosed an official figure. It also demonstrated improved performance compared to its predecessors (Open et al. 2023). However, this still many orders of magnitude smaller than the human brain’s estimated 100–1000 trillion synaptic connections.3 GPT-4 is a multimodal large language model that can generate responses to both text and images. In the ChatGPT interface, GPT-4 is integrated with DALLE 3, enabling it to comprehend a much broader range of nuances and details than earlier versions. In March 2024, Claude 3 Opus by Anthropic was released, boasting multimodal capabilities in generating images, tables, graphs, and diagrams. Moreover, Anthropic claims that Claude 3 Opus outperforms GPT-4 in generating human-like dialog and contextually aware responses. These rapid advances have, in turn, led the creative industries to face significant challenges. For example, DMG Media, the Financial Times, and Guardian Media Group have highlighted concerns about the potential impact on print journalism, particularly if AI tools reduce the need for users to click through to news websites, affecting advertising and subscription revenues (Communications and Digital Committee 2024). There is also concern about ‘AI-generated slop’–low-quality, mass-produced content created by AI that often lacks coherence or originality.4 It is typically used for spam, search engines, or clickbait, and is criticized for cluttering the internet and undermining genuine human-created content.

The generation of videos is significantly more challenging for AI than generating images. In February 2024, Google announced Gemini 1.5, which had the capability to process approximately 8 times more data than GPT-4; this comparison refers to a 1 M-token context window versus 128k tokens. This opening opportunities for video and audio processing.5 In the same month, OpenAI provided its first preview of Sora, a model capable of generating impressive realistic videos up to 1 min long. Based on the videos released by OpenAI, Sora appears to outperform other text-to-video models. Sora is currently available to ChatGPT subscribers, but new Plus accounts may face a wait-list or monthly credit limits. A month later, Gemini 1.5 announced its support for native audio understanding in 180 countries. With the emergence of these tools, together with the prospect of further advances, it is clear that video content creation will be a major beneficiary. This will further open up the media landscape for creativity and provide more opportunities for diverse storytellers, while also reducing production time. A recent example is the AI-generated Christmas commercial by Coca-Cola6, which elicited mixed responses in the press (Yang 2025), thereby pre-empting questions regarding potential hype. Such advertisements overcome the limitations of current technologies by using very short videos with rapid scene transitions, ensuring that any artifacts, such as unnatural fingers, are less apparent.

For the case of post-production workflows, generative AI may not have a direct impact, but the neural networks originally proposed for generative AI have been widely adapted to serve this purpose. This has led to significant improvements in both output quality and computational speed. Moreover, there is a noticeable trend towards adopting a unified framework rather than addressing individual tasks, as it better reflects real-world scenarios. For instance, natural history filmmaking involves challenging acquisition environments and high production standards. Filming often takes place in low light conditions, in the presence of heat haze, underwater or in adverse weather conditions. This often results in increased noise levels, focus issues, low contrast, color balance problems, and blurriness in the footage. In such cases, unified models can offer advantages in generalizing to diverse tasks and providing flexibility. Take Painter by BAAI Vision (Wang et al. 2023a) as an example, which employs an image pair as a task prompt (similar to a text prompt in LLMs), their model transfers the input image to produce a similar output as the task prompt, enabling it to undertake various tasks such as segmentation, low-light enhancement or rain removal.

While generative AI can facilitate and accelerate the creation and post-processing of digital media, there is an equivalent need to transmit or stream it efficiently to users. Although AI-based solutions have been proposed both for the enhancement of conventional video coding tools and for new compression frameworks, they are yet to be adopted in practical applications due to hardware constraints, complexity issues and a lack of standardization. Despite this, the latest learning-based video codecs have already demonstrated their potential to compete with conventional standardized video codecs and are being actively investigated in various standards bodies such as MPEG and AOM.

Furthermore, in recent years, AI has also impacted our ability to assess and monitor the perceptual quality of visual media. Advances have included new model architectures based on different attention mechanisms and the application of LLMs, which evidently improve model generalization. New training methodologies have also been proposed based on weakly/unsupervised learning, which address issues associated with the limited availability of labeled training content.

One of the exciting aspects of using LLMs in the creative sector is that ‘The human in the loop’ (Chung 2021) is simplified through text prompts, with sophisticated, multilingual language capabilities enabling artists to convey complex emotions and narratives. This is important because generative AI does produce mistakes, known as hallucinations. Human oversight is thus essential to correct this through reinforcement learning with feedback (Wu et al. 2023).

In this paper, the objective is to present the major technological advancements that have emerged since our previous review on AI in the creative industries (Anantrasirichai and Bull 2022). Whereas the earlier paper was written at a time when AI tools primarily served supportive roles, this updated review captures the disruptive shifts driven by generative AI and related technologies over the past few years. Adopting an integrative narrative rather than a systematic approach, we aim to synthesize major developments and identify cross-domain themes, rather than conduct an exhaustive quantitative survey. Sources published between 2022 and mid-2025 were selected from leading AI conferences (CVPR, ICCV, ECCV, NeurIPS, ICLR, IEEE TPAMI, IEEE TIP), verified arXiv preprints highlighting emerging developments, and industry releases, based on their technical significance and impact (e.g., OpenAI, Google DeepMind, Meta, Adobe). Selection was guided by recency, originality, and relevance to creative-industry applications, ensuring representative coverage of key models and trends. In summary, we initially screened more than 1000 documents published between 2022 and mid-2025, including major AI conference papers (60%), journal articles (30%), and industry or policy reports (10%). From these, around 450 sources were retained for detailed analysis across creation, enhancement, immersive media, and AI-driven production workflows. The review first outlines recent advances in AI technologies (Sect. 2), then highlights their applications in creative domains (Sect. 3), and finally discusses emerging challenges and future directions (Sect. 4).

Current advanced AI technologies

This paper provides a review of AI in the creative industries, building on our previous publication in 2022 (Anantrasirichai and Bull 2022). The reader is referred to that work for an introduction to AI, basic neurons, convolutional neural networks (CNNs), generative adversarial networks (GANs), recurrent neural networks (RNNs) and deep reinforcement learning (DRL). In this paper, we emphasize four key technologies that have grown in importance since 2022 that have had a significant impact on the creative industries. These are Transformers, Large language models (LLMs), Diffusion Models (DMs), and Implicit Neural Representations (INRs). It is important to note that, while these newer technologies are gaining prominence, those from previous generations remain in widespread use, often in conjunction with the newer ones. For instance, CNNs complement transformers since CNNs effectively capture local features and semantic meaning, while the attention mechanism in transformers captures global dependencies.

One important class within AI that has become dominant since our previous review comprises Foundation models (FMs). These were described by The Stanford Institute for Human-Centered Artificial Intelligence in 2021 (Bommasani et al. 2021) as “any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks”. Foundation models have been enabled by rapid advances in AI-oriented computing power and have underpinned the emergence and success of Large Language Models, particularly following the launch of ChatGPT by OpenAI in 2022. ChatGPT has become the fastest-growing consumer software application in history.7 These technologies are expanded on below.

Transformers

In 2017, Google AI introduced the concept of ‘Transformer’ architectures in their publication ‘Attention Is All You Need’ (Vaswani et al. 2017). This work has since been instrumental in the development and success of large language models alongside many other applications, including vision understanding (Dosovitskiy et al. 2021), and multiple modality learning (e.g., Gato (Reed et al. 2022)).

Before the advent of transformers, natural language processing (NLP) was performed using recurrent neural networks (RNNs), processing data sequences sequentially. In contrast, the ability of transformers to capture long-range dependencies through self-attention mechanisms that extend across all words in the sequence meant that the importance of different words could be established globally, understanding relationships regardless of their positions. This context-aware representation enables parallel processing of the entire sequence, making the transformers computationally efficient. A set of several attention layers running in parallel is called Multi-Head Attention.

The Transformer architecture, shown in Fig. 1a, comprises Encoder and Decoder sections, similar to many CNN-based generators. However, the encoder is now a stack of identical layers, concatenating a multi-head self-attention mechanism and a fully connected feed-forward network. The decoder is also a stack of identical layers, in which each layer has an additional sub-layer to perform multi-head attention over the output of the encoder stack.

Fig. 1.

Generative AI. a Transformer architecture (Vaswani et al. 2017). b The top row represents the diffusion process and the bottom row represents the generation process of the new image (Yang et al. 2023). c Latent Diffusion Models (LDM) (Rombach et al. 2022)

Mathematically, the attention function is computed from inputs: query Q, keys K, and values V. The matrix of outputs of attention function is

1

where is a dimension of K. The term is Dot-Product Attention, which yields a high similarity value when the two words are closely related. If Q and K are from the same sentence, Eq. 1 refers to self-attention, but if Q and K are from different sentences, it is referred to as cross-attention. Within the network, multi-head attention is actually employed to concurrently process attention and enable the model to collectively focus on information from distinct representation subspaces at various positions through the learnable parameters .

2

It should be noted that attention modules are not solely used in transformers, but have also been successfully integrated into other deep learning architectures such as CNNs, used for image classification (Li et al. 2022), object detection (Woo et al. 2018), and other computer vision tasks (Guo et al. 2022).

In 2020, the first successful training of a transformer encoder for image recognition was published (Dosovitskiy et al. 2021), referred to as a Vision Transformer (ViT). The ViT decomposes an input image into patches, similar to words in a sentence, and processes them through multi-head attention. Additionally, a Multilayer Perceptron (MLP) is employed as the feedforward network. In later work Microsoft introduced a hierarchical division of image inputs and a shifted window approach in their Swin Transformer (Liu et al. 2021). This was reported to outperform ViT by 2.4% in ImageNet-22K classification (21,841 different categories). Its version 2 (Liu et al. 2022) applied a cosine function in the attention module. enabling the scaling of capacity and resolution. More detail on transformer-based object detection is discussed in Sect. 3.4.2. To date, Swin Transformers have been widely adopted in a range of applications including image restoration (Fan et al. 2022). Comprehensive surveys on the use of transformers for image and video processing can be found in (Khan et al. 2022) and (Selva et al. 2023), respectively.

Transformers have been widely used and offer better performance across many tasks. One reason for this widespread adoption has been the availability of open-source Transformer libraries such as Hugging Face,8 a platform that assists developers to build applications for tasks including computer vision, NLP, audio, tabular data, multimodal tasks, and reinforcement learning. The platform also provides access to model zoo9 pretrained networks and datasets.

In recent years, state space models (Gu and Dao 2024; Zhu et al. 2024), commonly known as ‘Mamba’ have emerged. These are a linear variant of Transformers distinguished by their linear complexity in attention modeling. They are acknowledged to offer an equivalent or better performance than traditional Transformers, while demanding fewer computational resources and less memory.

Large language models

LLMs are based on transformer models using self-attention mechanisms as their core modules. Training comprises two steps: (i) pre-training with large amounts of unlabelled text data in an unsupervised manner to learn word meanings and relationships, and (ii) task adaptation through either fine-tuning or prompt-tuning. This pre-training approach is why OpenAI refers to their model as a Generative Pre-trained Transformer (GPT).

Fine-tuning involves training the model on new datasets, which must be large enough to ensure generalization to new tasks. Prompt-tuning and prompt engineering are emerging disciplines focused on developing and optimizing prompts for efficient model use. Prompts guide how AI models interpret and respond to user queries. Prompt engineering structures text or phrasing to steer the model toward desired outputs, relying heavily on experimentation and understanding model behavior. Prompt-tuning instead trains a small set of parameters before using the LLM, requiring relatively little new data. This approach converts text inputs into task-specific virtual tokens while keeping the pre-trained model unchanged (Lester et al. 2021). Its main drawback is reduced interpretability, though the paradigm has extended to other domains such as visual prompt tuning (Jia et al. 2022). For a comprehensive survey of LLMs, see Zhao et al. (2023).

To date, there are many LLM platforms as shown in Fig. 2. Figure 2a shows their timeline. Many surveys and evaluations of LLMs are also available (Zhao et al. 2023; Chang et al. 2024; Yao et al. 2024). These include FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets) Ye et al. (2024) which evaluates LLMs based on 12 fine-grained skills for comprehensive language model evaluation: logical correctness, logical robustness, logical efficiency, factuality, commonsense understanding, comprehension, insightfulness, completeness, metacognition, conciseness, readability, and harmlessness. Evaluation results from FLASK are shown in Fig. 2b.

Fig. 2.

a Timeline of large language models. b Performance comparison evaluated by FLASK (Ye et al. 2024)

A clear divide exists between open-source and proprietary LLMs. Open-source models such as LLaMA 3, Mistral, and Falcon promote transparency, reproducibility, and creative customization, allowing researchers to fine-tune or deploy them locally with improved data control. Proprietary systems like GPT-4, Gemini 1.5, and Claude 3, meanwhile, deliver stronger multimodal integration, safety alignment, and reliability–benefiting from access to large, private datasets and compute resources. Yet their closed architectures limit interpretability and independent evaluation. Together, these ecosystems form a complementary landscape balancing openness and innovation against performance and safety, shaping accessibility and creative autonomy in AI-driven practice.

Diffusion models

A generative model, in the context of AI, exploits machine learning to learn a probability distribution of the training data to generate new data samples. The very first models were based on Autoencoders (AEs) that learn to encode input data into a lower-dimensional representation (latent space) and then decode it back to its original form. A specific type of AE, a variational autoencoders (VAE) (Kingma and Welling 2014), learns the latent space as statistical parameters of probabilistic distributions, leading to significant improvement of the generated results. Concurrently, Goodfellow et al. (2014) introduced an alternative architecture known as a Generative Adversarial Network (GAN). GANs comprise two competing AI modules: a generator, which creates a sample, and a discriminator, which determines whether the received sample is real or generated. When comparing VAEs to GANs, VAEs exhibit greater stability during training, whereas GANs excel at producing realistic images. More details about AEs and GANs for creative technologies can be found in our previous review (Anantrasirichai and Bull 2022).

An important factor in driving the rapid growth of generative AI has been the development of diffusion probabilistic models (referred to as diffusion models (DMs)). The first DM was introduced in 2015 by Sohl-Dickstein et al. (2015), using Nonequilibrium Thermodynamics. However, it took a further 5 years for DMs to generate desirable results: the era of DMs began with Denoising Diffusion Probabilistic Models (DDPMs) proposed by Ho et al. (2020) in 2020 and Score-based diffusion models proposed by Song et al. (2021) in 2021. These involve a simplified process using a denoising autoencoder to approximate Bayesian inference. In brief, the models leverage a diffusion process to learn a probability distribution of the input data. As the name suggests, the data is diffused by gradually adding noise at each iteration step as shown in Fig. 1b. A deep neural network (DNN) is then trained to remove this noise, called the denoising process or reverse process. Consequently, the trained model uses random noise to generate data with characteristics similar to those of the training samples. Comparing to GANs, DMs provide higher diversity samples (Dhariwal and Nichol 2021) and a training process that is much more stable and does not suffer from mode collapse. DMs are however computationally intensive and require longer training times compared to GANs. The complexity can be significantly reduced by training the DMs in latent space. Latent Diffusion Models (LDM) (Rombach et al. 2022) use pretrained networks to convert images to feature maps, and perform training on a low-dimensional space. The diagram of LDM is shown in Fig. 1c.

Generating a synthesized sample at random might not be particularly useful, especially for creative industry applications. Therefore, conditional diffusion models have been proposed, supporting a wide range of applications such as text-to-sound, text-to-images, and image-to-videos. For DMs, the conditional distributions are modeled using a conditional denoising autoencoder. Classifier guidance was introduced in Dhariwal and Nichol (2021) to improve the generation of images of a desired class. For example, when we provide the model with information, such as ‘a flower’, the DM will synthesize a variety of flower images, as the word ‘flower’ guides the model toward the latent distribution that is formed by various images of flowers. The work in Choi et al. (2021) simply refines the latent space of well-trained unconditional DDPM so that the higher-level semantics of the synthetic samples are similar to the reference (conditioning). The LDM (Rombach et al. 2022) offers more flexible conditional image generators by adding cross-attention layers (referred to Transformers in Sect. 2.1) to the denoising autoencoder. A survey on the methods and applications of DMs prior to 2024 can be found in Cao et al. (2024).

Implicit neural representations

Implicit Neural Representations (INR), also called neural fields, neural implicits or coordinate-based neural networks, represent input content implicitly through learned functions F, as shown in Eq. 3. They can be considered as fields x (represented by a scalar, vector, or a tensor with a value, such as a magnetic field in physics) that are fully or partially parameterized by a neural network , typically an MLP (Xie et al. 2022).

3

Although this concept appears complex, the process is actually very straightforward. For example, in the case of an image, the coordinates of each pixel (x, y) contain color information (r, g, b). The INR inputs (x, y) to the MLP and learns to provide the output (r, g, b). The weights and biases of the MLP now represent such an image. Usually, the number of parameters of the MLP is smaller than the total number of pixels multiplied by 3, accounting for the 3 color channels. Hence, one of its emerging applications is in data compression (Kwan et al. 2024). Moreover, the INR can handle complex and high-dimensional data efficiently, attracting attention for visual computing applications such as 3D scene reconstruction.

Traditional MLPs employ ReLU (rectified linear unit) for non-linear activation due to its simplicity. However, Sitzmann et al. (2020) demonstrated that periodic functions, such as sinusoids, are more suitable for representing complex natural signals, offering a better fit to the first- and second-order derivatives of the signals. However, this activation can cause ringing artifacts. Saragadam et al. instead proposed using complex Gabor wavelets (Saragadam et al. 2023), which learn to represent high frequencies better and simultaneously are robust to noise.

One of the fastest-growing areas that exploits INRs is Neural Radiance Fields (NeRF), evidenced by 57 papers presented at CVPR, the largest annual conference in computer vision, in 2022 growing to 175 papers in 2023,10 before dropping to 71 in 2024, largely due to competition from 3D Gaussian Splatting.11 First introduced in 2020 by Mildenhall et al. (2020), NeRF is a form of neural rendering, a subset of generative AI, that generates novel views of a scene based on a partial set of 2D images. It achieves this by learning a mapping from 3D spatial coordinates and view directions to colors and density . This implicit representation allows NeRF to handle complex scenes with varying geometry and appearance, resulting in highly realistic renderings that include accurate lighting, shadows, and reflections. More detail can be found in Sect. 3.5.2.

Advanced AI for the creative industries

Similarly to our previous (2021) review of AI for the creative industries (Anantrasirichai and Bull 2022), Table 1 categorizes applications and corresponding AI-based solutions. These areas are explored in more detail below.

Table 1.

Creative applications and corresponding AI-based methods mentioned in this paper

Application		Technology
		Trans./Attn.	Diffusion	INR
Creation	Text	Vaswani et al. (2017); Wang et al. (2022); Open et al. (2023); Wei et al. (2024)
	Audio/music	Alayrac et al. (2022); Huang et al. (2022)	Li et al. (2022); Yang et al. (2023); Evans et al. (2025)
	Image	Alayrac et al. (2022); Esser et al. (2024)	Rombach et al. (2022); Brooks et al. (2023); Gal et al. (2023); Gandikota et al. (2024); Lian et al. (2024); Esser et al. (2024); Ren et al. (2024); Feng et al. (2025); Liu et al. (2025)
	Animation/video	Hong et al. (2023); Villegas et al. (2023); Azadi et al. (2023); Yu et al. (2023); Liu et al. (2023); Wang et al. (2024); Xu et al. (2024); Corona et al. (2024); Gupta et al. (2024); Hu (2024); Zhu et al. (2024)	Singer et al. (2023); Molad et al. (2023); Wang et al. (2023b); Wu et al. (2023); Liu et al. (2023); Gupta et al. (2024); Zhu et al. (2024); Wang et al. (2025); Wu et al. (2025)
	3D/AR/VR	Yang et al. (2024)	Xu et al. (2023); Melas-Kyriazi et al. (2023); Qian et al. (2024); Tang et al. (2024)	Tang et al. (2024); Ren et al. (2023); Zhao et al. (2024)
Information analysis	Text categorization	Sun et al. (2023); Shi et al. (2023); Hou et al. (2023); Ai et al. (2025)
	Film analysis	Mao et al. (2023); Krugmann and Hartmann (2024); Hartmann et al. (2023)
	Content retrieval	Metzler et al. (2021); Yan et al. (2023); Lu et al. (2023); Rajput et al. (2023); Li et al. (2024a, b)	Jin et al. (2023)
	Intelligent assistants	(King et al. 2024)
Content	Enhancement	(Xu et al. 2022; Liang et al. 2022; Wang et al. 2023c; Lin et al. 2024; Youk et al. 2024; Liang et al. 2024)	Hou et al. (2023); Yi et al. (2023); Jiang et al. (2023); Lin et al. (2024)	Yang et al. (2023)
Enhancement and post	Style transfer	(Deng et al. 2022; Moon et al. 2023; Chung et al. 2024)	Zhang et al. (2023); Chai et al. (2023)	Moon et al. (2023); Kim et al. (2024)
	Super-resolution	Liang et al. (2021); Lu et al. (2022); Liu et al. (2022); Chen et al. (2023); Li et al. (2023); Kang et al. (2023); Liang et al. (2024); Xu et al. (2024); Wang et al. (2025)	Saharia et al. (2023); Moliner et al. (2023); Gao et al. (2023); Cao et al. (2025); Wang et al. (2025)	Chen et al. (2021); Saharia et al. (2023); Fei et al. (2023); Gao et al. (2023); Yin et al. (2023)
Production	Restoration	Wang et al. (2022); Zamir et al. (2022); Li et al. (2023); Yang et al. (2023); Liang et al. (2024); Morris et al. (2025); Liang et al. (2021); Fan et al. (2022); Yu et al. (2022); Wang et al. (2023a); Song et al. (2023); Xu et al. (2023); Mao et al. (2022); Zhang et al. (2024); Zou and Anantrasirichai (2024); Fang et al. (2025); Yue et al. (2025); Jin et al. (2025); Yue et al. (2025); Shi et al. (2025); Liu and Anantrasirichai (2025)	Jiang et al. (2023); Fei et al. (2023); Yang et al. (2023); Nair et al. (2023); Jaiswal et al. (2023); Cao et al. (2025); Feng et al. (2025)	Jiang et al. (2023)
	Inpainting	Li et al. (2022); Liu et al. (2022); Ren et al. (2022); Zhou et al. (2023); Huang et al. (2024)	Moliner et al. (2023); Fei et al. (2023)
	Fusion	Ma et al. (2022); Rao et al. (2023); Liu et al. (2023); Li and Wu (2024)	Zhao et al. (2023)
	Editing/VFX	Shi et al. (2024)	Shi et al. (2024); Guo et al. (2024)
Information	Segmentation	Cheng et al. (2022); Kirillov et al. (2023); Ke et al. (2023); Wang et al. (2023a, d); Zou et al. (2023); Oquab et al. (2024); Ravi et al. (2024); Zhang et al. (2025)	Wu et al. (2023); Xu et al. (2023); Gu et al. (2024)	Gong et al. (2023); Cen et al. (2023)
Extraction and understanding	Recognition	Carion et al. (2020); Dosovitskiy et al. (2021); Zhu et al. (2021); Liu et al. (2021); Neimark et al. (2021); Liu et al. (2022); Huang et al. (2022); Oquab et al. (2024); Zhao et al. (2024); Im et al. (2025); Tian et al. (2025)	Li et al. (2023); Chen et al. (2023); Zhang et al. (2025); Wu et al. (2025)
	Tracking	Meinhardt et al. (2022); Zeng et al. (2022); Cui et al. (2022); Mayer et al. (2022); Yang et al. (2023); Chen et al. (2023); Zhang et al. (2023); Yi and Anantrasirichai (2024); Kang et al. (2025)	Luo et al. (2024); Xie et al. (2024); Zhang et al. (2024)	Jung et al. (2023)
	3D reconstruction	Wang et al. (2021); Zhang et al. (2023); Chen et al. (2023); Yang et al. (2024); Oquab et al. (2024); Yang et al. (2024); Liu et al. (2025)	Barron et al. (2022); Ji et al. (2023); Wynn and Turmukhambetov (2023); Ke et al. (2024)	Mildenhall et al. (2020); Pumarola et al. (2020); Müller et al. (2022); Barron et al. (2022); Mildenhall et al. (2022); Fang et al. (2022); Guo et al. (2022); Hu et al. (2022); Liu et al. (2024); Azzarelli et al. (2023); Zhan et al. (2024); Tang et al. (2024); Liu et al. (2025), Fridovich-Keil et al. (2023); Kerbl et al. (2023); Wu et al. (2024); Yu et al. (2024); Huang et al. 2024); Wang et al. (2025); Junkawitsch et al. (2025); Kong et al. (2025)
Compression		Zhu et al. (2022); Zou et al. (2022); Liu et al. (2023)	Careil et al. (2023); Yang and Mandt (2023); Hoogeboom et al. (2023); Ghouse et al. (2023)	Sitzmann et al. (2020); Dupont et al. (2021, 2022); Strümpler et al. (2022)
	Video	Xiang et al. (2022); Mentzer et al. (2022)	Li et al. (2024)	Chen et al. (2021); Bai et al. (2023); Kwan et al. (2024); Kim et al. (2024); Leguay et al. (2024); Kwan et al. (2024); Gao et al. (2024); Ruan et al. (2024); Kwan et al. (2024)
Quality		Cheon et al. (2021); Golestaneh et al. (2022); Shi et al. (2024)
Assessment		Wu et al. (2022); Feng et al. (2024); Wu et al. (2023); He et al. (2024); Peng et al. (2024)

Trans./Attn. include transformers, mamba and CNN-based architectures that use attention module

Some diffusion models employ the transformer in their denoising autoencoders

These methods are based on explicit neural representations

It is noted that for some compression and quality assessment tasks, there are other dominant network architectures in existing works. For example, LLMs have been used for image and audio compression, and visual quality assessment. Many neural audio codecs are also based on VQ-VAE models

Content creation

Content creation is a fundamental activity of artists and designers and the term ‘AI art’ refers to artforms created with the assistance of an AI algorithm or entirely by an AI system. This can refer to various digital forms including images, texts, audio, and videos. The roots of AI art can be traced back to the 20th century, exemplified by AARON, a computer program initiated in 1972 to autonomously produce paintings and drawings (Shapiro and Eckroth 1987). The practicality of AI art has been enhanced with advancements in deep learning, particularly GANs from 2014 and, more recently, transformers, DMs and INRs.

Text generation, script and journalism

In the era of LLMs, AI writing tools have been widely used to assist various writing tasks, including generation of written articles, blog posts, essays, and reports. These tools go beyond mere grammar and spelling checks; they boast advancements enabling them to analyze the style and tone of written material, adding images, videos and tables, offering suggestions to enhance clarity, coherence, and overall readability (Ippolito et al. 2022). Moreover, AI tools extend their utility beyond content generation by automating tasks like keyword generation, meta tags, and descriptions, thereby increasing search rankings using search engine optimization (SEO). Additionally, they support the process of publishing across multiple online platforms. Transformers have been used to generate image captions by combining information from the images with a word prefix or questions (Wang et al. 2022).

AI script generators serve as beneficial aids for writers, filmmakers, and game developers, offering inspiration, idea generation, and assistance in crafting entire scripts (Jeary and Gajjar 2024; Azzarelli et al. 2025). Human-AI brainstorming is helpful and saves time (Guo et al. 2024). Presently, there are numerous software and websites providing both free and paid script generation services. However, many of these tools are still constrained when it comes to longform creative writing. Dramatron, developed by Google (Mirowski et al. 2023), introduces hierarchical language generation, enabling the creation of cohesive scripts and screenplays spanning long ranges. This includes elements such as titles, characters, story beats, location descriptions, and dialogue.

As discussed earlier, chatbots are now powered by LLMs, effectively simulating human conversation. These fundamental LLMs are specialized for specific tasks. For instance, journalist AI and blog AI writers12 generate content with layouts suitable for print or online publication. Additionally, AI tools exist that are designed to detect AI-generated content (e.g., for checking for copyright), AI-writing styles, content originality, and to ensure the naturalness and flow of articles. Undoubtedly, generative AI is reshaping the way artists and journalists operate. For an in-depth exploration of the impact and implications of these technological advancements on news organizations, refer to the survey conducted by Beckett and Yaseen (2023).

Generating text and scripts automatically can also be done through image and video inputs without text prompts (e.g., image captioning Stefanini et al. 2023) and with text prompts. These approaches are referred to as Vision Language Models (VLMs): multimodal models that learn from images and text. The most common and prominent models often consist of an image encoder, an embedding projector to align image and text representations, often via a dense neural network, and a text decoder stacked in this order. The most well-known technique is Contrastive Language-Image Pre-training (CLIP) (Radford et al. 2021). More recent work in Wei et al. (2024) scales up the vision vocabulary by incorporating new image features into the existing CLIP model, resulting in improved content understanding. A comprehensive survey of VLMs for vision tasks can be found in Zhang et al. (2024).

Audio and music generation

Similar to language models, AI-based music generation has rapidly advanced due to unsupervised learning on large datasets and the use of transformers (see Sect. 2.2). Examples of such systems include MuseNet,13 Magenta Studio,14 and Musicfy.15 These tools assist in music composition by learning complex musical patterns, predicting the next word or music note in a sequence, and mixing specified instruments. Moreover, AI tools can convert one type of sound into another, such as from whistling to violin or from flute to saxophone.16 This capability is invaluable for artists who may not be proficient in playing all the instruments they wish to incorporate, saving both time and costs. In 2024, Suno has released a model capable of producing radio-quality music that can be created in 2 min.17 Later, Udio18 was launched. This offers a prompt to create lyrics and music with a maximum duration of 90 s, and also appears to have, at least some, awareness of copyright.

AI voice software changes vocalizations from one person to another, for example enabling users to train the model to convert other people’s voices into their own, e.g., lalals,19 Kits,20 Media.io,21 etc. Certain software, such as Voice.ai,22 even offers real-time voice changing capabilities. The technologies behind this use a transformer to learn voice features and patterns in mel-spectrogram form. For example, the framework proposed in Yang et al. (2023) uses a DM-based method with a transformer backbone to turn text input into a mel-spectrogram using the vector quantized variational autoencoder (VQ-VAE) (Oord et al. 2017). Next, this mel-spectrogram is transformed into a sound wave. Unlike a regular spectrogram, the mel-spectrogram is based on the mel-frequency scale, which offers higher resolution for lower frequencies. Voice style transfer often uses zero-shot learning (a model is trained to recognize classes or categories that it has never encountered during training) (Huang et al. 2022) or few-shot learning (a model trained with only one or a few examples per class) (Wang et al. 2022). Stable Audio Open (Evans et al. 2025) introduces a text-conditioned generative model for non-speech audio, trained on Creative Commons licensed data, capable of producing state-of-the-art 44.1kHz stereo audio.

Another emerging AI technology application is in the field of spatial audio. In 2022, Apple Music revealed that, in just over a year, more than 80% of its worldwide subscribers were enjoying the spatial audio experience, with monthly plays in spatial audio increasing by over 1000%.23 With head tracking, this technology significantly enhances the immersive experience. Masterchannel has launched SpatialAI,24 claiming it to be the world’s first spatial mastering AI. This processes audio files and returns an optimized track for streaming platforms, along with an individually optimized stereo version for traditional distribution. All these advancements leverage transformer-based technologies.

Image generation

As described in Sect. 2.3, recent advances in AI technologies for image generation are based on Diffusion Models (DMs). Well-known and highly competitive text-to-image models include Stable Diffusion,25 Midjourney,26 DALLE,27 and Ideogram.28 Released in June 2024, the latest version of Stable Diffusion (SD3), has been reported to outperform state-of-the-art text-to-image generation systems such as DALLE 3 (released August 2023) (Esser et al. 2024), Midjourney v6 (released December 2023), and Ideogram v1 (released February 2024) in terms of typography and prompt adherence, based on human preference evaluations. These open-source tools are built on a Multimodal Diffusion Transformer (MM-DiT) architecture, which integrates attention from both text and images. LLM4GEN (Liu et al. 2025) fuses features from LLM and CLIP models to enhance the semantic understanding in text-to-image diffusion models, enabling them to better handle complex and dense prompts involving multiple objects. Examples of text-to-image generation are shown in Fig. 3a comparing the performance of four models, i.e. Ideogram v1, DALLE 3, Photoshop 2025, and sdxy-turbo by Nvidia. It is clear that hands are one of the most difficult features to generate, e.g., one hand has six fingers.

Fig. 3.

Text-to-image generation (generated on 27 November 2024). a text-to-image generation by Ideogram v1, DALLE 3, Photoshop 2025, and sdxy-turbo by Nvidia. b The top-row images were generated by DALLE in ChatGPT 4. The bottom-row images are generated by LLM-grounded Diffusion (Lian et al. 2024)

DALLE 3, available on ChatGPT 4, also provides an inpainting tool, allowing the user to manually select the area to edit. However, as of April 2024, its performance is still limited. As illustrated in Fig. 3b, the selected area is the white car, and with the follow-up request to change the white car to the red car, DALLE 3 generates correctly. However, if asked to replace it with a bicycle, it does not work. LLM-grounded Diffusion (Lian et al. 2024) was the first to introduce a framework that allows multiple rounds of user requests without the need for manual selection on the image. This is achieved by generating layout-grounded images, first using stable diffusion and then masking the latent variables as priors for the next round of generation29. Since then, text-driven image editing has seen significant improvements in quality, with most recent approaches adopting Diffusion Transformer architectures (Feng et al. 2025; Huang et al. 2025).

Similar to DALLE 3, Photoshop features a Generative Fill tool30 designed to generate new images or assist with photo editing. It accepts a text prompt and provides several generation choices. After defining the editing area, users can remove and add new objects (more inpainting tasks are discussed in Sect. 3.3.5), transfer to new styles, and expand content within images. Recently, Brooks et al. introduced InstructPix2Pix (Brooks et al. 2023), a conditional diffusion model that generates image editing examples without predefined editing areas. By combining GPT-3 and Stable Diffusion, the model effectively captures and matches the semantic meaning of the content in both text and image. Sometimes, style and context are not easy to describe in words. Textual Inversion (Gal et al. 2023) personalizes large pre-trained text-to-image diffusion models based on specific objects and styles, using 3-5 images of a user-provided concept. ByteDance announced Hyper-SD (Ren et al. 2024) which proposed trajectory segmented consistency distillation and provides real-time high-resolution image generation from drawing with a control text prompt.

Video generation and animation

Despite the success of text-to-image generation, text-to-video generation has not advanced at the same pace, growing more rapidly only in 2024 due to its computational expense and content complexity. Several major companies and private platforms have now released offerings, including Gemini 1.5 by Google, Make-A-Video by Meta, and Sora by OpenAI. Make-A-Video (Singer et al. 2023), through a spatiotemporally factorized diffusion model, leverages joint text-image priors and super-resolution in space and time, though some results exhibit flickering artifacts.31 Gen-2 by Runway32 supports both text- and image-to-video generation, producing smooth 4-second clips. In April 2024, Adobe Premiere Pro announced integration of generative AI tools for video extension with third-party models by OpenAI, Runway, and Pika Labs,33 including contextual selection, inpainting for object removal, and object addition to videos via text prompts.

Text-to-video technologies, combined with AI voice, have been tested not only by artists and producers but also by a wider audience. Results from these experiments–such as automatically turning scripts into trailers and music videos–have been widely shared online.34 However, scene composition and transitions still require further editing to meet production needs.35 In April 2024, Microsoft introduced VASA-1 (Xu et al. 2024), which converts a single image and speech audio clip into a realistic video of talking faces mimicking expressions and head movements (Fig. 4, right). The resulting quality surpasses Google’s VLOGGER (Corona et al. 2024), which uses a similar diffusion-based approach but additionally generates upper-body and hand motion. Recently, ByteDance proposed an audio-driven interactive head-generation model (Zhu et al. 2024) offering listening and speaking states during multi-turn conversations, based on a conditional diffusion transformer.

Fig. 4.

(Left) Video-to-3D post animation by DeepMotion. (Right) Image and audio to video by VASA-1 (Xu et al. 2024)

The main technologies underpinning text-to-video and image-to-video tasks are based on diffusion models (DMs) combined with 3D convolutions–or separate spatial and temporal convolutions–and attention modules (Wang et al. 2023b). Tune-A-Video (Wu et al. 2023) modifies the style of an input video using text prompts, leveraging pretrained text-to-image models and attention tuning for temporal consistency. Early methods often exhibited flickering, as observed in the CVPR2023 text-guided video editing competition. Dreamix (Molad et al. 2023) mitigates this but produces blurry videos. CogVideo (Hong et al. 2023) employs VQ-VAE to convert frames into tokens fused with text embeddings to generate new videos. Phenaki (Villegas et al. 2023) uses transformers for variable-length outputs, though with lower quality than DMs. Comprehensive evaluations appear in Liu et al. (2023). More recent work applies spatiotemporal layers to model dynamics (Gupta et al. 2024), redesigning transformer blocks for latent video diffusion with restricted spatial and spatiotemporal attention. LaVie (Wang et al. 2025) shows that simple temporal self-attention, combined with rotary positional encoding, effectively captures temporal correlations. Image-to-video generation is analogous to text-to-video but conditions diffusion models on images rather than text; hybrid approaches combine textual descriptions (for motion) and images (for scene layout) (Wu et al. 2025). An increasing number of free and commercial tools are emerging, including Veo 3 by Google DeepMind,36 Kling AI,37 Pika 2.2,38 and Hailuo AI.39 Though not perfect, their generated videos appear remarkably realistic.

Generating characters with human posture and motion from text prompts has also become popular. Make-An-Animation (Azadi et al. 2023) trains on image-text datasets and fine-tunes on motion capture data, adding additional layers to model the temporal dimension. Animate Anyone by Alibaba Group (Hu 2024) inputs a real photo or anime of a person with a sequence of guided poses. The results are significantly better than existing techniques, including Disco (Wang et al. 2024) and Bidirectionally Deformable Motion Modulation (BDMM) (Yu et al. 2023). They also suggest using Animate Anyone with Outfit Anyone40 to produce a character with a reference outfit.

Viggle41 claims to be the first video-3D foundation model embodying an actual understanding of physics. It combines a character and a text prompt about motion to generate character animation. Available AI tools for 3D on the market include DeepMotion42 that offers text-to-3D post animation and video-to-3D post animation, shown in Fig. 4 (left). The later function can track multiple people from real video and generate replicated characters with the same motions.

Augmented, virtual and mixed reality, and 3D content

While the benefits of LLMs in Augmented Reality (AR) directly target educational purposes, enhance cognitive support, and facilitate communication (Xu et al. 2025), mixed reality (MR) has once again become exciting since the release of the Apple Vision Pro in February 2024. This demonstrated the potential of MR experiences by merging real-world environments with computer-generated ones. Thanks to the rapid growth of AI-based 3D representation (see Sect. 3.5), the generation of AR/VR/MR content has advanced significantly. Real-time rendering with immersive interaction has improved, and real scenes can now be generated avoiding uncanny valley effects. There has also been an attempt to use autoregressive and generative models to estimate lighting, achieving a visually coherent environment between virtual and physical spaces in AR (Zhao et al. 2024).

Similar to other content generation tools, LLMs have been influenced on immersive technologies, including text-to-3D and image-to-3D. Exciting examples include Holodeck (Yang et al. 2024), which automatically generates 3D embodied environments via text-prompt interactions with a large language model (GPT-4). 3D objects are gathered from Objaverse (Deitke et al. 2023), a dataset with 800K+ annotated 3D objects. RealFusion (Melas-Kyriazi et al. 2023), a single image to 3D object generator, merges 2D diffusion models with NeRF, improving Instant-NGP (Müller et al. 2022), which provides an API for VR controls. NeuralLift-360 (Xu et al. 2023) also uses diffusion models to generate priors for novel view synthesis. Magic123 (Qian et al. 2024) is the latest image-to-3D tool that uses 2D and 3D priors simultaneously to produce high-quality high-resolution 3D geometry and textures. DreamGaussian (Tang et al. 2024) offers text-to-3D and image-to-3D by adapting 3D Gaussian splatting (more in Sect. 3.5.3) into generative settings using a diffusion prior. This generates photo-realistic 3D assets with explicit mesh and texture maps within only 2 min. DreamGaussian4D (Ren et al. 2023) employs image-to-video diffusion and a 4D Gaussian Splatting representation to generate an image-to-4D model. The results are not very sharp, but they can be further edited with Blender.

In July 2024, Shutterstock launched its Generative 3D service in commercial beta, powered by NVIDIA Edify, a multimodal generative AI architecture. This service enables creators to rapidly prototype 3D assets and generate 360-degree HDRi backgrounds to light scenes using text or image prompts. In conjunction with OpenUSD, the created scenes can be rendered into 2D images and used as input for AI-powered image generators, allowing for the production of precise, brand-accurate visuals.

Information analysis

Text categorization

Applications of text categorization include detecting spam emails, automating customer support, monitoring social media for harmful content, etc. At its core, text categorization involves assigning predefined labels to text documents, which can be anything from a tweet to a lengthy article. LLMs are particularly well-suited for this task due to their ability to comprehend complex and nuanced language. One of the main advantages of using LLMs in text categorization is their transfer learning capability. Models can be pre-trained on a large amount of text and then fine-tuned on a smaller, task-specific dataset, with or without further post-processing techniques. For example, CARP (Sun et al. 2023) applies kNN to integrate a diagnostic reasoning process for final decision. ChatGraph, proposed by Shi et al. 2023, utilizes ChatGPT to refine text documents. It uses a knowledge graph, extracted using another specifically defined prompt, and finally, a linear model is trained on the text graph for classification. Multiple learners are also used to enhance the performance (Hou et al. 2023; Ai et al. 2025).

Advertisements and film analysis

Not only does AI assist in generating ideas and content, but it can also aid creators in effectively matching content to their audiences, particularly on an individual level (Feizi et al. 2023). This effectively helps in advertising personalization–eMarketer43 reported that nearly nine out of ten consumers are comfortable with their browsing history being utilized to create personalized ads. In contrast to outdated syntax-style searches, advanced LLM tools can comprehensively grasp user intent behind each search through conversation prompts, providing advertisers with a high level of granularity.

Current advances in generative AI would greatly benefit sentiment analysis, also known as opinion mining, where opinions are gathered from social media, articles, customer feedback, and corporate communication and are analyzed to understand the emotion of the owners. This is a potential tool for filmmakers and studios, enabling the creation of effective and targeted marketing campaigns. By analyzing viewer emotions and opinions, AI can provide valuable insights into audience preferences, aiding in the optimization of film marketing strategies. Sentiment analysis with modern generative AI produces more accurate results. Technically, LLMs learn complex patterns and relationships in text data for sentiment classification (Mao et al. 2023; Krugmann and Hartmann 2024). SiEBERT (Hartmann et al. 2023) provides a pre-trained model with open-source scripts to be fine-tuned to further improve accuracy for novel applications. Cinema Multiverse Lounge (Ryu et al. 2025), a multi-agent conversational system, allows users to interact with LLM-driven agents, each embodying a distinct film-related target user.

Content retrieval and recommendation services

Generative retrieval (GR) was pioneered by (Metzler et al. 2021). Unlike traditional retrieval, which adheres to the “index-retrieve-then-rank” paradigm, the GR paradigm employs a single model to obtain results from query input. The model generally involves deep-learning based transformers, generating output token-by-token. More recent work in Li et al. (2024b) introduces learning-to-rank training to enhance the performance system up to 30%. GR has several advantages including substituting the bulky external index with an internal index (i.e., model parameters), significantly reducing memory usage, and enabling optimization during end-to-end model training towards a universal objective for information retrieval tasks. Conversational question answering techniques have been integrated to enhance the document retrieval (Li et al. 2024a).

When retrieving visual content, recent work exploits generative models to enhance content-based model search (Lu et al. 2023). These models decode the text, image, or video query into samples of possible outputs, which are then used to learn statistics for better matching between the query and output candidates. DMs are also employed for visual retrieval tasks, where they learn joint data distributions between text queries and video candidates (Jin et al. 2023). A comprehensive survey on Generative Information Retrieval is available in (Li et al. 2025).

While the retrieval task involves users directly defining a specific query input, recommendation services operate by retrieving content based on previous usage patterns. Essentially, a recommendation engine is a system that suggests products, services, or information to users through data analysis. Research in Chua et al. (2023) has reported a positive association between buyers’ attitudes toward AI and their behavioral intention to accept AI-based recommendations, with potential for further growth. Notable examples include the recommendation framework developed by Google Rajput et al. (2023), which utilizes GR. This framework assigns Semantic IDs to each item and trains a retrieval model to predict the Semantic ID of an item that a given user may engage with. A report by Aggarwal (2025) states that the recommendation accuracy of recommendation services has increased from 45.0% to 91.5% with the integration of generative AI.

Intelligent assistants

Intelligent assistants refer to software programs or applications that use AI and NLP to interact with users and provide helpful responses or perform tasks. These assistants can range from simple chatbots to sophisticated virtual agents capable of understanding and responding to complex queries. They’re designed to assist users in various tasks, from answering questions and providing information to scheduling appointments and controlling smart home devices.

Current LLMs obviously enhance the performance of intelligent assistants, designed to understand complex inquiries and generate more natural conversational responses, such as King et al. (2024). Generative AI can also be used to enhance the performance of human customer support agents, aiding in search and summarization, as discussed in the previous section. Brynjolfsson et al. (2023) examined the implementation of a generative AI tool designed to offer conversational guidance to customer support agents. Their research revealed that AI assistance significantly enhances problem resolution and customer satisfaction. Furthermore, they observed that AI recommendations prompt low-skill workers to adopt communication styles akin to those of high-skill workers. AI-based intelligent assistants may currently be more focused on educational purposes, but they can clearly help artists write more efficiently (Lee et al. 2024) or assist in customizing personal requirements (Sajja et al. 2024). The performance of personalized assistants can be enhanced with domain-specific knowledge to provide more in-depth responses to users (Jiang et al. 2025).

Content enhancement and post production workflows

Enhancement

In our previous review paper (Anantrasirichai and Bull 2022), we discussed AI technologies for contrast enhancement and colorization as separate topics, as methods were developed specifically for each task. However, in recent years, there has been a shift towards addressing more complex issues, such as those encountered in low-light environments and underwater scenarios. These real-world situations often involve a combination of challenges, including low contrast, color imbalance, and noise.

In low-light conditions, scenes often exhibit low contrast, leading to focusing difficulties or the need for long exposures, which can result in blurred images and videos. To address this, LEDNet (Zhou et al. 2022) has introduced a synthetic dataset for such scenarios and incorporated a learnable non-linear activation function within the network to enhance feature intensities. Meanwhile, SNR-Aware (Xu et al. 2022) estimates spatial-varying Signal-to-Noise Ratio (SNR) maps and proposes local and global learning branches using ResNet and transformer architectures, respectively. NeRCo (Yang et al. 2023) addresses the low-light problem with INR, which unifies the diverse degradation factors of real-world scenes with a controllable fitting function. Diffusion models (DMs) have also become popular choices for low-light image enhancement (HOU et al. 2023; Yi et al. 2023; Jiang et al. 2023). Diff-Retinex (Yi et al. 2023) formulates the low-light image enhancement problem into Retinex decomposition, and employs multi-path generative diffusion networks to reconstruct the normal-light Retinex probability distribution. A recent state-of-the-art approach presented in Jiang et al. (2023) decomposes images into high and low frequencies using wavelet transform. High frequencies are enhanced using a transformer-based pipeline, while the low frequencies undergo a diffusion process. This method achieves nearly 2.8dB improvement over the state-of-the-art transformer-based approach, e.g. LLFormer Wang et al. (2023c), and significantly better than INR-based method, NeRCo (Yang et al. 2023), on a real low-light image benchmarking dataset. The technique has been extended for video enhancement in Lin et al. (2024). The output of the enhancement typically depends on user preferences. This has been viewed as a one-to-many inverse problem, with attempts to solve it using Bayesian approaches. For example, a Bayesian Enhancement Model (BEM) (Huang et al. 2026) incorporates Bayesian Neural Networks (BNNs) to capture data uncertainty and produce diverse outputs. The method can be used with Transformers or Mamba as the architecture backbone.

Regarding video enhancement, transformer and DMs are still in their early stages. STA-SUNet (Lin et al. 2024) has demonstrated that using transformers for low-light video enhancement outperforms CNN-based methods (Anantrasirichai et al. 2024). The recent Mamba-based network (Huang et al. 2025) also demonstrates promising results, outperforming STA-SUNet by more than 2 dB in PSNR. It is important to note that low-light enhancement is subjective. While most training datasets use normal lighting conditions as ground truth (Lin et al. 2024), the enhanced images and videos may alter the mood and tone of the content. Therefore, the tools for creative industries should be adjustable, not only for entire images and videos but also adaptive to specific areas and content. For instance, CLE Diffusion (Yin et al. 2023) enables user-friendly editing of lighting with fine-grained regional controllability.

Recent efforts have focused on enhancing User-Generated Content (UGC) videos–authentic recordings created by individuals rather than brands, often showcasing real experiences with products or services. The winning solution of the NTIRE 2025 Challenge on UGC Video Enhancement (Safonov et al. 2025) implemented a pipeline of four sequential modules: color enhancement, denoising, BasicVSR++ restoration (Chan et al. 2022), and SwinIR (Liang et al. 2021). This method achieved a 17% higher subjective score than the second-place entry, which used a two-stage framework, highlighting a notable improvement in perceived visual quality.

Style transfer

Style transfer in AI art refers to a technique where the artistic style of one image (or video) is applied to another image (or video) while preserving the content of the latter. Style transfer has numerous applications in art, design, and image editing, allowing artists and designers to create unique and visually appealing compositions by blending different artistic styles with existing images (or videos). The applications also include image-to-image and sequence-to-sequence translations.

StyTr2 (Deng et al. 2022) is the first transformer-based method for style transfer, applying content as a query and style as a key of attention. InST (Zhang et al. 2023) utilizes Stable Diffusion Models as the generative backbone and introduces an attention-based textual inversion module to learn the description of the content. StableVideo (Chai et al. 2023) uses a text prompt to describe the desired appearance of the output, transforming the input video to have a new look based on a diffusion model. For instance, a video of a white car driving in summer can be altered to show a red car driving in winter. A large pre-trained DM is employed in Chung et al. (2024), where the style is injected to manipulate the self-attention of the decoder. To deal with the disharmonious color, they propose an adaptive instance normalization. A survey of style transfer using transformers and diffusion models can be found in Zhou et al. (2025). Implicit Neural Representations (INRs) are less commonly used in style transfer tasks due to the difficulty of modeling the cross-representation between style and content. (Moon et al. 2023) combined INRs with vision transformers for generalizable style transfer; however, the results remain limited in quality. In contrast, the method proposed by Kim et al. (2024) uses multilayer perceptrons (MLPs) to map image coordinates to the colors of the stylized output, guided by features extracted from both the content and style inputs to allow controllability.

Upscaling imagery: super-resolution (SR)

Impressive super-resolution (SR) results from transformer and diffusion models have been published extensively in the past few years. Originally, SR methods were developed using multiple low-resolution (LS) images, as different features in each image are combined to construct an enhanced one. However, these methods are not practical, as in most cases only one LS image is available. Hence, more methods have been developed for single image super-resolution (SISR).

The first use of a transformer, called ESRT, was for capturing long-term dependencies, such as repeating patterns in buildings. This was done in the feature domain extracted by a lightweight CNN module (Lu et al. 2022), outperforming those that use only CNNs. Since then, most SISR methods have been based on transformers. The Hybrid Attention Transformer (HAT) (Chen et al. 2023) was introduced, which improves the SR quality over ESRT by more than 2dB when upscaling 2–4. However, the NTIRE 2023 Real-Time Super-Resolution Challenge (Conde et al. 2023) showed that the winner, Bicubic++ (Bilecen and Ayazoglu 2023), uses only convolutional layers and achieves the fastest speed at 1.17ms in upscaling 720p to 4K images. This method is significantly faster than any of the participants in the NTIRE 2025 Challenge (Chen et al. 2025), where Transformer-based architectures continue to dominate as the mainstream approach.

For DMs, SR3 by Google (Saharia et al. 2023) has produced truly impressive results. It operates by learning to transform a standard normal distribution into an empirical data distribution through a sequence of refinement steps, interpolating in a cascaded manner–upscaling 4 at a time. Later, IDM (Gao et al. 2023) combines INR with a U-Net denoising model in the reverse process of the DM. It is crucial to emphasize again that DMs are generative models. The SR results are generated based on the statistics we provide to the model during training (LR training samples). This is not for a restoration task, but rather for synthetic generation. A survey in SISR using DMs can be found in Moser et al. (2024).

For video SR, numerous methods have emerged as part of a unified enhancement framework, as discussed in the previous section. One of the pioneering works to incorporate transformers specifically for video SR tasks is the Trajectory-aware Transformer for Video Super-Resolution (TTVSR) (Liu et al. 2022). Although the results are slightly inferior to those of BasicVSR++ (Chan et al. 2022), which employs CNN and was introduced around the same time, both methods significantly enhance detail and sharpness compared to previous approaches, albeit not in real time. To address this limitation, the Deformable Attention Pyramid (Fuoli et al. 2023) has been introduced, offering slightly lower quality but a speed-up of over 3. Recently, Adobe announced their VideoGigaGAN (Xu et al. 2024), which can perform 8 upsampling. This is achieved by adding flow estimation and temporal self-attention to the GigaGAN upsampler (Kang et al. 2023), which is primarily used for image SR, and text-to-image synthesis. Cao et al. (2025) introduce a zero-shot video super-resolution framework that leverages a pre-trained image diffusion model, and replaces the spatial self-attention layer with a novel short-long-range (SLR) temporal attention layer. Recently, SeedVR integrated text information (captions) into a Diffusion Transformer (DiT) model, achieving state-of-the-art performance in video super-resolution.

Compared to traditional upscaling methods, generative AI can add details that did not exist in the original input image. These methods excel at generating high-quality natural images and structures, such as buildings, which are commonly included in training datasets. However, the process can be slow and may produce unpredictable results if the input image has very low resolution or contains content rarely seen in natural images. As shown in Fig. 5 (left), generative AI fails to upscale the knitting texture areas, instead generating lines more commonly found in typical images. While AI methods produce sharper edges, they perform less effectively on text.

Fig. 5.

(Left) Examples of SR (4) using generative model. (Right) Real-time portrait editing with FacePoke

Restoration

In our previous review paper (Anantrasirichai and Bull 2022), we categorized the work on restoration into several different types of distortions, including deblurring, denoising, dehazing, and mitigating atmospheric turbulence. Recent work, however, uses a unified network architecture to address these as inverse problems , where x and y are the ideal and observed data, respectively. h is a degradation function, such as blur, and n is additive noise. Often, the super-resolution task is also considered as an inverse problem, meaning h includes the downsampling process. Note that although designed as a single network, the model is trained with each distorted dataset separately.

The pioneering transformer-based method for image restoration, SwinIR (Liang et al. 2021), employs several concatenated Swin Transformer blocks (Liu et al. 2021). SwinIR surpasses state-of-the-art CNN-based methods proposed up to the year 2021 in super-resolution and denoising tasks. The model is smaller and reconstructs fine details more effectively. Other two popular approaches that emerged in the same timeframe are Uformer (Wang et al. 2022) and Restormer (Zamir et al. 2022). Both incorporate Transformer blocks into hierarchical encoder-decoder networks, employing skip connections similar to those in U-Net. Their objective was to restore noisy images, sharpen blurry images, and remove rain. The networks focused on predicting the residual R and obtaining the restored image through . While their performance is very similar, Restormer has half the parameters of Uformer. More recently, GRL by Li et al. (2023) exploits a hierarchy of features in a global, regional, and local range using different ways to compute self-attentions as an image often show similarity within itself in different scales and areas, which outperforms SwinIR and Restormer. Additionally, Fei et al. introduced the Generative Diffusion Prior (Fei et al. 2023) for unsupervised learning, aiming to model posterior distributions for image restoration and enhancement. VmambaIR (Shi et al. 2025) incorporates Mamba blocks into the U-Net architecture, achieving superior performance compared to SwinIR and Restormer in both visual quality and model size.

For video restoration, the general framework comprises frame alignment, feature fusion and reconstruction. The process could be similar to image restoration but input multiple frames and run through the sequences in a sliding window manner to exploit the temporal information within a number of consecutive frames. Recently, Video Restoration Transformer (VRT) (Liang et al. 2024) and its improved version with a recurrent process (RVRT) (Liang et al. 2022), have emerged as the state of the arts for video super-resolution, deblurring, denoising, and frame interpolation. This method introduces temporal reciprocal self-attention in the transformer architecture and parallel warping using MLP. These innovations enable parallel computation and outperform the previous state-of-the-art methods by up to 2.16 dB on benchmark datasets. FMA-Net (Youk et al. 2024) proposed multi-attention for joint video super-resolution and deblurring, achieving fast runtime with nearly 40% improvement over RVRT, and the restored quality was reported better by up to 3%.

For audio restoration, most software discussed in Sect. 3.1.2 offers tools for enhancing audio quality, such as eliminating background noise, echo, microphone rumble, and occasionally room reverberation, which have been well-established even before the advent of deep learning. There have been efforts to utilize AI for learning global contextual information to aid in the removal of unwanted sounds, leading to better final quality (Yu et al. 2022). The latest advancements in this domain are primarily focused on addressing issues where significant portions of the audio data are missing. For instance, Moliner et al. (2023) tackle problems such as audio bandwidth extension, inpainting, and declipping by treating them as inverse problems using a diffusion model. For a comprehensive survey on the use of diffusion models in restoration tasks, refer to (He et al. 2025).

The following methods have been proposed for specific problems, but ideally, they should be adaptable for other tasks, even though they may not perform as well as they do for the original task.

• i)Deblurring: A lightweight deep CNN model was recently proposed in (Pan et al. 2023), where a new discriminative temporal feature fusion has been introduced to select the most useful spatial and temporal features from adjacent frames. Feature propagation along the video is done in the wavelet domain. The deblurring performance is comparable to RVRT (Liang et al. 2022), but it is 5 times faster. DaBiT (Morris et al. 2025) mitigates focal blur content with depth information and applies SR for further enhancing fine details. Note that not only in software, but AI technologies have also been integrated into hardware. This includes autofocus, which is crucial for capturing sharp images of subjects, especially in dynamic environments where manual adjustments are impractical due to rapid movement. AI-driven autofocus methods have emerged, often tailored for specific camera hardware. For instance, Choi et al. proposed an autofocus model optimized for dual-pixel Canon cameras (Choi et al. 2023). Additionally, Yang et al. investigated the correlation between language input and blur map estimation, utilizing semantic cues to enhance autofocus performance (Yang et al. 2023). Remarkably, their model achieves comparable results to previous state-of-the-art methods while being more lightweight (Yang et al. 2023). Autofocus could be used in conjunction with real-time object tracking (see Sect. 3.4.3) to produce desirable sharpness for moving objects in the video. Recently, Feng et al. (2025) proposed a novel residual diffusion deblurring framework that integrates a conditional diffusion model guided by a defocus map and incorporates residual learning into the single-image defocus deblurring process.
• ii)Denoising: SUNet (Fan et al. 2022) applies Swin transformer blocks combined in a UNet-like architecture. Denoising with diffusion models (DMs) (Yang et al. 2023) has been proposed by diffusing with estimated noise that is closer to real-world noise rather than Gaussian noise, achieving better performance than SwinIR (Liang et al. 2021) and Uformer (Wang et al. 2022). INR with complex Gabor wavelets as activation functions show promising denoising results (Saragadam et al. 2023). The NTIRE 2025 Image Denoising Challenge (Sun et al. 2025) revealed that the top-performing methods combined transformer-based and convolutional network architectures. Similarly, recent advances in video denoising also adopt a hybrid approach that integrates both architectures (Jin et al. 2025; Yue et al. 2025).
• iii)Dehazing: Vision transformers for single image dehazing were proposed in DehazeFormer (Song et al. 2023). Similar to SUNet, it is a UNet-like architecture, but introduces Rescale Layer Normalization for better suit on improving contrast. The Fast Fourier Transform (FFT) has been employed in Fang et al. (2025) due to the phase spectrum conveying more structural detail than the amplitude spectrum and demonstrating greater robustness to contrast distortion and noise. Then cross-attention between the RGB and YCbCr color spaces is applied. This approach achieves nearly 5 dB higher PSNR than DehazeFormer on a real-world smoke dataset. For video dehazing, Xu et al. (2023) introduced a recurrent multi-range scene radiance recovery module with the space-time deformable attention. They also employ physics prior to inform haze attenuation. This method outperforms DehazeFormer by approximately 1 dB.
• iv)Mitigating atmospheric turbulence: Similar to dehazing, physics-inspired models have been widely developed to remove turbulence distortion (Jaiswal et al. 2023; Jiang et al. 2023), while complex-valued CNNs have been proposed to exploit phase information (Anantrasirichai 2023). There was also an attempt to use instance normalization (INR) to solve this problem, offering tile and blur correction (Jiang et al. 2023). However, diffusion models outperform on a single image (Nair et al. 2023), and transformer-based methods remain state-of-the-art for restoring videos (Zhang et al. 2024; Zou and Anantrasirichai 2024; Liu and Anantrasirichai 2025). Mamba architecture, employed in Hill et al. (2025), outperforms Transformers and improves object detection performance. Work in Hill et al. (2026) has shown that an object detector trained together with mitigation improves the restoration results. A recent review can be found in Hill et al. (2025).

Inpainting

Visual inpainting is the process of filling in lost or damaged parts of an image or video. CNNs and GANs have already achieved impressive results (see our previous review paper Anantrasirichai and Bull 2022). Recent work has focused more on editing rather than simply filling in the missing areas. This means users can now mask large areas of an image, and AI tools generate multiple results for users to choose from, a technique known as pluralistic inpainting (Zheng et al. 2019). Some notable methods include the following: Mask-Aware Transformer (MAT) (Li et al. 2022) offers several outputs to fill a large missing area, consisting of a convolutional head, a transformer body, and a convolutional tail for reconstruction, along with a Conv-U-Net for refinement. PUT (Liu et al. 2022) proposes a patch-based vector VQ-VAE and unquantized Transformer to minimize information loss. Spa-former (Huang et al. 2024) employs a UNet-like architecture, where each level performs transformer with sparse self-attention to remove coefficients with low or no correlation, leading to memory reduction, while improving result quality by up to 5% compared to PUT.

Video inpainting presents greater complexity compared to image inpainting, despite the abundance of information available in an image sequence. The process typically involves tracking masks across frames, estimating optical flow, and ensuring temporal consistency. The current state-of-the-art methods include DLFormer (Ren et al. 2022) and ProPainter (Zhou et al. 2023). DLFormer conducts inpainting in latent space and utilizes discrete codes for video representation. On the other hand, ProPainter employs flow-based deformable alignment to enhance robustness to occlusion and inaccurate flow completion. The method excels in filling complete and rich textures, achieving a speed of 12 fps for full HD video. Video inpainting is also used for dubbing. DINet (Zhang et al. 2023) replaces the mouth area to synchronize with a new language being spoken.

A comprehensive survey of learning-based image and video inpainting, covering approaches such as CNNs, VAEs, GANs, transformers, and diffusion models, can be found in Quan et al. (2024). Additionally, Elharrouss et al. (2025) provide an in-depth review of the current challenges and future directions specific to transformer-based inpainting techniques.

Image fusion

Image fusion is the process of merging multiple images from either the same source (such as varying focal points or exposures) or different modalities (e.g. visible and infrared cameras) into a single image. This process integrates complementary information from the various images to enhance overall quality, improve interpretation, and increase the usability of the final image.

Transformers and CNNs have been combined to extract global and local information, respectively. Most methods use CNNs for feature extraction, with transformers operating in the latent space (Ma et al. 2022; Rao et al. 2023). Notable methods include SwinFusion (Ma et al. 2022), which utilizes a self-attention-based intra-domain fusion unit and a cross-attention-based inter-domain fusion unit to achieve multi-modal and digital photography image fusion. Transformer-based image fusion has also been applied to downstream tasks like segmentation (Liu et al. 2023), achieving superior results by leveraging the additional information. Self-attention blocks are employed to enhance intra-feature representations, while the cross-attention mechanism integrates inter-feature information to improve the quality of the fused output (Li and Wu 2024).

DDFM, the first diffusion model-based image fusion method, estimates noise in the reverse process by combining multiple inputs (Zhao et al. 2023). The expectation-maximization (EM) algorithm is integrated to estimate the noise distribution parameters, resulting in sharper images compared to traditional DDPM. For an in-depth review, the reader is referred to recent work in (Karim et al. 2023), (Zhang and Demiris 2023).

Editing and visual special effects (VFX)

Editing or modifying specific areas of an image is much easier with DM technologies, particularly for headshot photos, such as targeting the eyes and mouth on the face (Guo et al. 2024). This capability has been extended to video generation (see Sect. 3.1.4). Figure 5 shows an example of the online tool, FacePoke,44 which allows users to move the head and modify the shapes of the eyes and mouth in real time. Motion-I2V (Shi et al. 2024) provides motion blur and motion drag tools to control specific areas of an image to add motion. The method is based on a diffusion-based motion field predictor and motion-augmented temporal attention.

VFX aims to create and/or manipulate imagery outside the context of a live-action shot in filmmaking and video production. When adding objects, scenes, and effects into traditional photographic videos, generative AI has obviously become an important tool, but some manual operations are still required. For example, in After Effects (EA),45 the user selects the area where the object will be added and uses text prompts to describe such object. Subsequently, with the current EA version, the user will need to apply motion tracking so the generated object is moved accordingly.

AI technologies can upscale, enhance, and restore low-quality or old footage. For example, standard definition videos can be converted to high definition or even 4K quality without traditional manual remastering processes. This is particularly useful for remastering old movies or enhancing visual details in scenes. Generative AI has also simplified and accelerated automated processes, such as rotoscoping (Tous 2024), an animation technique where animators trace over motion picture footage frame by frame to create realistic action. AI models can accurately detect and segment objects and characters in video frames, significantly speeding up the post-production process. Additionally, AI can assist the rapid creation of 3D models from 2D images generating realistic animations with minimal input data, facilitating complex human motions and synchronized facial expressions to voiceovers. One restriction is that current technologies still cannot yet generate full 4K accurate visual effects.

Information extraction and understanding

AI plays a crucial role in automating and optimizing the process of information extraction and understanding, enabling organizations to derive actionable insights from large and diverse data. Yan et al. (2023) have categorized information extraction tasks based on the Format-Time-Reference space, as illustrated in Fig. 6a, where object detection and video object segmentation (VOS) are considered to be the simplest and the most complex tasks, respectively. Recent advancements in this field draw significant inspiration from LLMs. These advancements include the utilization of prompts as conditional inputs for acquiring information. Moreover, following the pipeline approach used in LLMs, there is a growing trend towards leveraging very large datasets to pre-train models before fine-tuning them for specific downstream tasks. For instance, Meta AI (Oquab et al. 2024) has introduced DINOv2, aimed at enriching information about visual content through self-supervised learning. This model was trained with 142 million carefully selected images, employing the ViT architecture. Google have introduced VideoPrism (Zhao et al. 2024), a tool for scene understanding including classification, localization, retrieval, captioning, and question answering (QA). The model was trained on an extensive and diverse dataset consisting of 36 million high-quality video-text pairs and 582 million video clips accompanied by noisy or machine-generated parallel text.

Fig. 6.

a Tasks in Object-centric understanding defined by Yan et al. (2023) (REC Referring Expression Comprehension, RES Referring Expression Segmentation, VOS Video Object Segmentation, RVOS Referring Video Object Segmentation, MOT Multiple Object Tracking, MOTS Multi-Object Tracking and Segmentation, VIS Video Instance Segmentation, SOT Single Object Tracking. b Current high-quality segmentation (Ke et al. 2023)

Segmentation

The need for segmentation has grown dramatically in the past few years, given its central role in visual perception. Many segmentation methods now integrate an input prompt for users to define their preferred output appearances, such as pixel-wise segmentation, bounding boxes around objects, or segmented areas of interest. Most of these methods utilize transformer architectures (Cheng et al. 2022). Among them, Segment Anything (SAM) by Meta AI (Kirillov et al. 2023) stands out as a pioneer in promptable segmentation approaches. This method computes masks in real-time and has been trained with over 1 billion masks across 11 million images, facilitating transferability from zero-shot to new image distributions and tasks. HQ-SAM (Ke et al. 2023) enhances SAM by incorporating global–local feature fusion, leading to high-quality mask predictions. SegGPT (Wang et al. 2023d) proposed context ensemble strategies and allows users to tune a prompt for a specific dataset, scene, or even a person, while SEEM (Zou et al. 2023) provides a completely promptable and interactive segmentation interface. More recently, SAM2 (Ravi et al. 2024) introduced support for real-time video segmentation. It is a unified model trained on a larger dataset than SAM. Interactive tools enable users to mark areas of interest and specify regions to exclude from the segmentation map. Zhou et al. propose an audio-visual segmentation (AVS) to generate pixel-level segmentation masks for sounding objects in audible videos. DVIS++ by (Zhang et al. 2025) introduces a universal video segmentation framework capable of producing instance, semantic, and panoptic segmentation outputs. This transformer-based architecture comprises a segmentor, tracker, and refinement module, achieving state-of-the-art performance across several video segmentation benchmarks.

With DMs tehchnologies, Baranchuk et al. (2022) have investigated semantic representation, and found DMs outperform other few-shot learning approaches. DiffuMask Wu et al. (2023) automatically generate image and pixel-level semantic annotation using pre-trained Stable Diffusion with input as a text prompt. It has been proven that using these synthetic data improve segmentation accuracy. Currently, the state-of-the-art panoptic segmentation is the method developed by Nvdia, which is based on text-to-image DMs (Xu et al. 2023), outperforming the previous methods by up to 7.6%.

Applying INRs to segmentation is more popular in the medical domain, as the specific signals used, such as computed tomography (CT) and magnetic resonance imaging (MRI), can be formulated as continuous functions. In creative technologies, unsupervised domain adaptation (UDA) and INRs are used for continuous rectification function modeling in Gong et al. (2023), achieving superior segmentation results in night vision. Recently, this work has been integrated with a non-local means block in Lin et al. (2025) showning a significant improvement for instant segmentation in low-light scenes.

3D segmentation is also crucial for scene manipulation. In radiance fields, earlier segmentation methods required additional modules such as using k-means clustering to separate objects from the background (Goel et al. 2023). However, the recent SA3D approach (Cen et al. 2023) segments 3D objects using NeRFs as the structural prior. SA3D operates by taking a trained NeRF and a set of prompts from a single view, then performing an iterative procedure. This involves rendering novel 2D views, self-prompting SAM for 2D segmentation, and projecting the segmentation back onto 3D mask grids. A comprehensive survey of 3D segmentation in computer vision can be found in He et al. (2025).

Detection and recognition

Introduced in 2020, DETR by Facebook AI (Carion et al. 2020) was one of the first to adopt a transformer architecture for object detection. The approach achieves comparable results to an optimized Faster R-CNN (Ren et al. 2017), introduced in 2015. Deformable convolution has alson been used, (Deformable DETR Zhu et al. 2021), resulting in training faster with approximately 5% accuracy improvement. A survey until 2022 (Zou et al. 2023) reported that Deformable DETR and Swin Transformers (Liu et al. 2021) outperform pure CNN-based YOLOv4 (Bochkovskiy et al. 2020). SwinV2 improves the first version by replacing original dot product attention with scaled cosine attention, improving accuracy by approximately 5%. Later, RT-DETR (Zhao et al. 2024) improved inference speed by decoupling the intra-scale interaction and cross-scale fusion of features with different scales. RT-DETR is 25% faster than YOLOv846 with 6% improvement on MS COCO Object Detection dataset. Recently, YOLOv10 (Wang et al. 2024) has been released. YOLOv10 further improves the speed of detection approximately by 30% over RT-DETR with the same accuracy. A review of transformer-based methods for object detection can be found in Li et al. (2023), Kheddar (2025). Recently, YOLO12 (Tian et al. 2025) introduced an attention-centric architecture, achieving a 2.1% and 1.2% mAP improvement over YOLOv10-N and YOLOv11-N respectively, with only a slight decrease in speed.

To detect 3D objects, the transformer-based method MonoDTR (Huang et al. 2022) incorporates depth estimation from a single 2D image (Yang et al. 2024) to predict 3D bounding boxes. More 3D object detection methods have been developed for autonomous driving (Song et al. 2024); however, these approaches can also be adapted for AR and VR applications (Im et al. 2025).

While DMs are primarily used to generate synthetic datasets (Wu et al. 2023; Fang et al. 2024), they have also been demonstrated to function as zero-shot classifiers by Li et al. (2023). DMs are also of interest for detection tasks, Although feature extractors are still predominantly based on CNNs, such as ResNet, or Transformers (like Swin). DiffusionDet (Chen et al. 2023) formulates object detection as a denoising diffusion process from noisy boxes to object boxes, reporting performance that surpasses DETR. DMs have also been employed for anomaly detection (Zhang et al. 2025; Wu et al. 2025), functioning similarly to zero-shot classifiers.

Tracking

Object tracking stands out as one of the tasks that greatly benefits from transformers since attention is needed in both space and time. An experimental survey cited in Kugarajeevan et al. (2023) reveals that transformer-based methods consistently rank at the top of the leaderboard across various datasets. In the Visual Object Tracking (VOT) challenges of 2023,47 all of the top-10 employed transformer-based methodologies. The highest-performing approach achieved a 10% improvement in tracking quality compared to the winner in 2020. The current state-of-the-art for single-object tracking,48 however, is based on cross-attention and Mamba (Kang et al. 2025).

The first three tracking-by-attention approaches are TrackFormer (Meinhardt et al. 2022), MixFormer (Cui et al. 2022), and ToMP (Mayer et al. 2022). TrackFormer extracts visual features using a CNN-based encoder, which are then tracked using a vanilla transformer (Vaswani et al. 2017) in a frame sequence, while MixFormer introduces cross-attention between the target and search regions. ToMP tracks the objects using prediction aspects. Many more methods have been proposed, including SeqTrack (Chen et al. 2023) and Track Anything Model (TAM) (Yang et al. 2023). SeqTrack extracts visual features with a bidirectional transformer, while the decoder generates a sequence of bounding box values autoregressively with a causal transformer. TAM combines SAM (Kirillov et al. 2023) and XMem (Cheng and Schwing 2022), offering tracking and segmentation performance on the human-selected target. However, the masked area is still not very sharp, and there is a subtle degree of temporal inconsistency. MOTRv2 (Zhang et al. 2023) combines YOLOX (Ge et al. 2021) for object recognition and MOTR (Zeng et al. 2022) for tracking, outperforming TrackFormer by 20%. Additionally, some methods have been specifically proposed for challenging environments, such as low light (Yi and Anantrasirichai 2024) and small objects, as seen in AnyFlow (Jung et al. 2023). The latter exploits INR to upsample a continuous coordinate-based flow map, similar to SISR technique proposed in Chen et al. (2021).

Similarly to detection tasks, DMs for tracking tasks are used as downstream processes by concatenating the diffusion head to the feature extraction backbone. However, a spatial-temporal fusion module has been added to the diffusion head to exploit temporal video features (Luo et al. 2024). DiffusionTrack (Xie et al. 2024) localizes the target in a progressive diffusion manner, which is claimed to better handle challenging scenarios. The method in (Zhang et al. 2024) exploits spatial-temporal weighting to suppress the probability of the tracker changing the target to the distractors. It, however, reports underperformance compared to MixFormer.

3D reconstruction and rendering

Bridging the gap between digital and physical realms, 3D reconstruction and rendering are integral to various creative technologies. In film and animation, they enable the creation of detailed digital models that blend seamlessly with live-action footage. Video games and digital twins leverage these technologies for dynamic environmental rendering. VR and AR use 3D reconstruction to create immersive and interactive experiences, with AR integrating digital content into real-world contexts. With recent AI technologies, 3D reconstruction and rendering have become faster and closer to reality. In particular, neural radiance fields and Gaussian Splatting enable artists and film producers to create shots that cannot be done in the real shooting environments.

Depth estimation

Accurate depth information (alongside texture data) is typically required to construct 3D models. Depth sensors, such as lidar (Light Detection and Ranging) and structured-light 3D scanners, can be used for this purpose, but their applications are often limited by distance and cost. Consequently, vision-based sensors have become widely used. These sensors utilize two or more cameras to simulate human binocular vision or employ a single camera to capture images from different locations.

As deep learning can capture monocular cues such as object size, texture gradients, and perspective, depth estimation from a single image can produce accurate results. There have been attempts to use transformers, such as Zhang et al. (2023) and Chen et al. (2023), and diffusion models, such as Ji et al. (2023) and Ke et al. (2024). Amongst these, Depth Anything v2 (Yang et al. 2024) has become a state-of-the-art monocular depth estimation method. It is built on the previous version (Yang et al. 2024), jointly trained on large-scale labeled and unlabeled images and uses semantic priors from pretrained encoders. Depth Anything v2 significantly outperforms V1 in fine-grained details and robustness by using synthetic images and pseudo-labeled real images, as well as by extracting intermediate features from DINOv2 (Oquab et al. 2024), which is trained with vision transformers. One of the notable capabilities of Depth Anything v2 is its ability to predict the depth of transparent and reflective surfaces.

Neural radiance fields

Neural Radiance Fields (NeRFs), introduced in Mildenhall et al. (2020), have demonstrated the ability to learn a 3D scene from a smaller number of images captured from various viewpoints, as opposed to photogrammetry. They excel in neural rendering, particularly in view-dependent novel view synthesis, and have effectively tackled several challenges associated with automated 3D capture (Xie et al. 2022), such as accurately representing the reflectance properties of the scene. NeRFs offer high-resolution photo-realistic novel views and flexibility in postprocessing. They have hence gained significant attention in cinematography (Azzarelli et al. 2024), as they offer reduced time and cost, particularly for outdoor shooting.

In the NeRF process (see Fig. 7a), the camera positions and orientations are typically estimated from a series of 2D images using techniques like feature-mapping and Structure-from-Motion (SfM), as demonstrated in Schönberger and Frahm (2016). Leveraging INR, each image (or camera pose) is mapped into camera rays that traverse the scene, generating 3D points with directional radiance (towards the camera). These points are then processed by an MLP to predict volume density and emitted radiance. Subsequently, volume rendering techniques are employed to generate an image, which is compared with the original via loss calculation. The MLP iteratively refines the model by minimizing this loss.

Fig. 7.

3D representation. a Neural Radiance Fields (NeRFs) (Mildenhall et al. 2020). b Gaussian Splatting (Kerbl et al. 2023). c Example scenes of VR-GS system for 3D content interaction in VR (Jiang et al. 2024)

Since their introduction, there have been many variants of NeRFs aimed at improving their performance. Mip-NeRF360 (Barron et al. 2022) proposed an unbounded anti-aliased technique achieving full 360 degree content. Google Research (Mildenhall et al. 2022) trains NeRF from noisy RAW images captured in the dark scene, allowing changing viewpoint, focus, exposure, and tone mapping simultaneously. With segmentation techniques significantly advanced (see Sect. 3.4.1), there have been integrations utilizing semantic segmentation to enhance 3D representation (Guo et al. 2022). DSEM-NeR (Liu et al. 2025) integrates the pretrained CLIP model to extract multimodal features–including color, depth, and semantics–from multi-view 2D images, thereby enhancing the reconstruction quality of complex scenes.

While the rendering quality of NeRF is very good, training and rendering times remain extremely high. The Instant-NGP tool developed by Nvidia Müller et al. (2022) enables real-time training of NeRFs by bypassing sampling in empty spaces and dense areas, and by incorporating multi-resolution hash encoding techniques. These advancements substantially reduce the computational burden associated with representing high-resolution image features – training times have been reduced from hours to just a few seconds. Moreover, it offers VR controls for immersive 3D rendering experiences using OpenXR.49 This allows users to navigate scenes, manipulate objects, and interact with the environment directly through VR headsets. Diffusion models are integrated to regularize NeRF reconstructions (Wynn and Turmukhambetov 2023), resulting in smoother depth continuity and clearer edges where depth discontinuities occur.

The initial application of NeRFs to dynamic scenes was undertaken by Pumarola et al. (2020), known as D-NeRF. However, the current leading method for generating high-quality novel views of real dynamic scenes is TiNeuVox (Fang et al. 2022). It enhances temporal information by interpolating voxel features before feeding them into the radiance network to estimate density and color, similar to ordinary NeRF. DynVideo-E (Liu et al. 2024) adds an MLP to predict motion fields but focuses on human-centric content. PaReNeRF (Tang et al. 2024) addresses large-scale dynamic scenes using patch-based sampling. The main drawback of these methods is the large model size and/or long training time. Therefore, K-planes (Fridovich-Keil et al. 2023) propose a simple planar factorization for volumetric rendering, achieving low memory usage (1000 compression over a full 4D grid). Wavelet transforms are employed in Azzarelli et al. (2023) to further reduce model size. KFD-NeRF (Zhan et al. 2024) incorporates a Kalman filter-guided deformation field for more accurate motion estimation.

3D Gaussian splatting

The main issue with NeRFs as a method to generate high-quality novel views is training time, which can exceed a day for high-resolution content on a single RTX 3090 GPU (Hu et al. 2022). 3D Gaussian Splatting (3D-GS) (Kerbl et al. 2023) has been introduced to address this, using anisotropic 3D Gaussians to form a high-quality, unstructured representation of radiance fields. The process estimates a sparse point cloud through SfM. Each point possesses 3D Gaussian properties, such as position, covariance matrix, opacity, and spherical harmonics coefficients representing colors. The optimization of these parameters is interleaved with steps that control the density of the Gaussians to better represent the scene, as shown in Fig. 7b. A survey of 3D-GS can be found in Fei et al. (2024).

In contrast to traditional NeRFs based on implicit scene representations, 3D-GS provides an explicit representation that can be seamlessly integrated with post-processing manipulations, such as animating and editing. VR-GS (Jiang et al. 2024) offers intuitive and interactive physics-based game-play with deformable virtual objects and realistic environments represented with 3D-GS. The example scenes are shown in Fig. 7c. Physics-inspired approaches are also integrated to improve 3D modeling in different media, such as 3D underwater scenes (Wang et al. 2025).

For dynamic scenes, 4D Gaussian Splatting (4D-GS) (Wu et al. 2024) introduces a Gaussian deformation field for motion and shape. It exploits a multi-resolution encoding method, achieving real-time rendering of up to 82 fps at a resolution of 800800 pixels on an RTX 3090 GPU. Instead of developing in 4D, CoGS (Yu et al. 2024) exploits 3D-GS by integrating control mechanisms in separate regions to learn individual temporal dimensions. SC-GS (Huang et al. 2024) extracts sparse control points and uses an MLP to predict time-varying 6 DoF transformations. While the results show better visual quality than 4D-GS and CoGS, the performance heavily relies on camera pose estimation. Kong et al. (2025) represent dynamic scenes using sparse, time-variant attribute modeling with a deformable MLP, while efficiently filtering out anchors corresponding to static regions. Their model achieves fast rendering speeds of over 110 FPS at a resolution of 960540–nearly 10 times faster than SC-GS–and delivers a 1 dB improvement in PSNR.

LUMA AI50 and Polycam51 offer free tools for Gaussian splatting and photogrammetry creation for non-commercial use. The 3D objects created can be experienced with VR headsets for more immersive 3D and further used or developed in other applications. However, these tools have limitations in handling dynamic scenes due to occlusions, sparse observations per timestamp, and object reappearances over time. Rendering dynamic avatars can produce higher quality results by incorporating additional information. For example, EVA (Junkawitsch et al. 2025) disentangles the 3D Gaussian appearance into skeletal motion, facial expressions, body movements, and skin. These components are then splatted to render the final photorealistic image.

Digital twins

A digital twin is a virtual replica of a physical object, system, or process, continuously updated with real-time data for purposes such as simulation, testing, monitoring, and maintenance. This technology is increasingly adopted across various applications within the creative industries. For example, in product design and branding, it enables immediate observation of how a design performs in various contexts, facilitating the development of user-friendly products. Unilever reported that integrating digital product twins with 3D technologies, such as NVIDIA Omniverse,52 enabled the creation of product imagery twice as fast and 50% more cost-effective.53 Digital twins also allow consumers to explore products or spaces virtually, simulating real-world interactions.

Accenture plc, a global professional services company, collaborated with Walt Disney Studios to develop digital twin technologies aimed at transforming the filmmaking process.54 Their goal is to generate remotely accessible 3D models, enabling virtual exploration of potential shooting locations without requiring physical visits. The Virtual StudioLAB provides a digital replica created using 360-degree imagery and 3D modeling. These innovations have streamlined pre-production workflows for major productions from Marvel Studios and 20th Century Studios.

Digital representations such as avatars, proxies, and digital twins are increasingly being explored in artistic contexts, particularly in relation to identity, presence, and embodiment in virtual environments. The Tate Modern’s film programme Avatars, Proxies and Digital Twins (Feb–May 2025) investigated these themes through curated audiovisual works, offering critical reflections on digital personhood. By engaging with diverse narrative forms, the programme highlighted the sociocultural implications of digital self-representation, prompting discourse on authenticity, agency, and the role of immersive media in shaping future human–machine interaction.

Data compression

Data compression plays an important role in the delivery of creative content to audiences, effectively reducing memory and bandwidth requirements during signal storage and transmission (Bull and Zhang 2021). Although coding methods based on conventional signal processing theories are still widely employed in most standards and application scenarios, learning-based solutions have emerged in research, showing great potential to achieve competitive performance in recent years. This subsection provides a brief overview of the recent advances in image, video, and audio compression, in particular focusing on the approaches proposed after 2021.

Image compression

Since the first neural image codec (Ballé et al. 2016) was proposed in 2016, numerous learning-based image compression methods have been developed, with significant performance improvements reported (Ballé et al. 2018; Cheng et al. 2020). Driven by the latest advances in neural network architectures, neural image codecs now outperform standard image codecs. Instead of using CNNs as the basic network structure, transformer-based architectures have become popular, offering the potential for better compression efficiency. Notable examples include SwinT-ChARM (Zhu et al. 2022), STF (Zou et al. 2022) and LIC-TCM (Liu et al. 2023). SwinT-ChARM (Zhu et al. 2022) employs Swin transformers for non-linear transforms and outperforms the latest standard image codec, the Versatile Video Coding (VVC) Test Model (VTM, All Intra). STF (Zou et al. 2022) is based on a symmetrical transformer framework containing absolute transformer blocks in both the down-sampling encoder and the up-sampling decoder, which also shows improved rate-quality performance over VTM. LIC-TCM (Liu et al. 2023) exploits the local modeling ability of CNN and the non-local modeling performance of transformers, and proposes a parallel transformer-CNN mixture block. This new network structure, together with a channel-wise entropy model based on attention modules using Swin transformers, contributes to the superior performance of STF, with more than 10% bitrate savings over VTM.

An alternative approach to learned image coding is based on advanced generative models. Early works (Agustsson et al. 2019; Mentzer et al. 2020) employed GANs to generate more photo-realistic results with improved visual quality. Although these models fail to outperform conventional, CNN-based or transformer-based approaches, when distortion-based quality metrics, e.g., PSNR, are used for performance evaluation, they have been reported to perform well when perceptual quality models, such as MS-SSIM (Wang et al. 2003) and VMAF (Li et al. 2018), or subjective tests are employed to measure perceived video quality. More recently, diffusion models have been applied in image compression to allow realistic reconstruction at ultra-low bitrates (Careil et al. 2023) achieving competitive performance compared to GAN-based models (Yang and Mandt 2023). However, it should be noted that some of these generative models aim to generate (or synthesize) images with “perfect realism” rather than reconstruct results that are most similar to the original content. Notable work in this category includes image codecs using score-based generative models (Hoogeboom et al. 2023) and the diffusion-based residual augmentation codec (DIRAC) (Ghouse et al. 2023). Moreover, another type of generative model based on INR has been employed for image compression; this learns a mapping between the spatial coordinates and the respective pixel values for the input image. The learned INR model is then compressed through parameter quantization and model compression to minimize the required bitrate. Notable INR-based image codecs include COIN/COIN++ (Dupont et al. 2021, 2022) and (Strümpler et al. 2022) that combine SIREN networks (Sitzmann et al. 2020) with positional encoding.

In order to evaluate and compare neural image codecs under fair test conditions, public grand challenges have been increasingly run, typically associated with international conferences. One of the most well-known of these is the Challenge on Learned Image Compression (CLIC) (Challenge on Learned Image Compression 2024). In its latest competition, the best performing learned image codec (Li et al. 2024), which is based on a GAN-enhanced Vector Quantized Variational AutoEncoder (VQ-VAE) framework, offered up to 0.6dB PSNR gain over VTM (version 22.2, All Intra) at similar bitrates; this codec is based on an autoencoder architecture with latent refinement and perceptual losses.

To support the deployment of neural image codecs, the International Organization for Standardization (ISO)/International Electrotechnical Commission(IEC) has developed a royalty-free learned image coding standard, denoted as JPEG AI (Ascenso et al. 2023), which aims to offer significant performance improvement over existing standards for both human and machine vision tasks. The Call for Proposals of JPEG AI was published in 2022, while the Working Draft and the Committee Draft outlining its core coding system were released in 2023 (Alshina et al. 2024), with its first version published in October 2024 (Alshina et al. 2024). JPEG AI follows the same framework (the auto-encoder structure) as most existing neural image codecs, and its test model JPEG AI VM (version 4.3) has been reported to achieve up to 28.5% coding gains over VVC VTM (All Intra mode) (Karabutov et al. 2024).

Video compression

Compared to image coding, the compression of video content is a much more challenging task, particularly for immersive video formats and diverse content types. Although video coding standards including H.264/AVC (Advanced Video Coding), H.265/HEVC (High Efficiency Video Coding) and H.266/VVC (Versatile Video Coding) are still predominant in real-world applications, learning-based video coding has advanced dramatically in the past five years, with new deep learning enhanced conventional coding tools and end-to-end optimized neural video coding frameworks proposed.

• i)The enhancement of conventional coding tools focuses on employing deep learning techniques to improve the performance of one or more coding modules in a standard-compliant codec. These modules include intra prediction (Li et al. 2021), inter prediction (Jin et al. 2021), in-loop filtering (Feng et al. 2024), post-decode filtering (Zhang et al. 2023) and resolution re-sampling (Wang et al. 2023). To facilitate efficient integration, the MPEG Joint Video Experts Team (JVET) built a test model in 2022 based on VTM 11, named Neural Network-based Video Coding (NNVC) (Li et al. 2023), with its latest version NNVC7.1 containing two major learning-based coding tools, neural-network based intra prediction and in-loop filtering, which has achieved an up to 13% coding gain over VTM 11 (Random Access mode) (Galpin et al. 2024). However, this learning-based codec requires much higher computational complexity (up to 477 kMACs/pixel) and high-spec GPU support compared to conventional codecs. Meanwhile, members of the Alliance of Open Media (AOM) have also developed multiple CNN-based coding tools for the next generation of video coding standard beyond AV1. The latest proposals focus on the trade-off between performance and complexity, with one of them based on in-loop filtering and super-resolution, which achieves an average BD-rate saving of 3.9% (in PSNR) over AVM, the test model of AV2, but only requires a much lower computational complexity (below 1.5kMACs/pixel) (Joshi et al. 2023). More recently, research has been conducted to further improve the performance of these learning-based coding tools utilizing more advanced network architectures, including ViTs (Kathariya et al. 2023), and diffusion models (Li et al. 2024). There are also investigations on applying preprocessing before compression (Chadha et al. 2021; Tan et al. 2024), where the training of the deep preprocessors is based on proxy video codecs and/or rate-distortion loss functions to simulate the behavior of conventional video coding algorithms.
• ii)End-to-end optimized neural video codecs. Alongside the enhancement of coding tools in conventional video codecs, more recent research activities have focused on using neural networks to implement the whole coding workflow, enabling data-driven end-to-end optimization. The performance of these neural video codecs has advanced significantly in the last 5 years, since the first attempt, DVC (Lu et al. 2019), was published. DVC matched the performance of a fast implementation of H.264 (x264). However currently, learned video coding algorithms (e.g., DCVC-FM (Li et al. 2024) and DCVC-LCG (Qi et al. 2024)) are able to compete or even outperform the state-of-the-art standard codecs, such as VVC VTM under certain coding configurations. These learning-based methods often focus on enhancement from different perspectives, including feature space conditional coding (e.g., FVC (Hu et al. 2021) and DCVC (Li et al. 2021)), instance adaptation (Khani et al. 2021; Oh et al. 2024), and motion estimation (e.g., DCVC-DC (Li et al. 2023)). New architectures have also been proposed such as CANF-VC (Ho et al. 2022) based on a video generative model, MTMT (Xiang et al. 2022) using a masked image modeling transformer-based entropy model and VCT (Mentzer et al. 2022) based on a video compression transformer. It is noted that, although promising coding performance has been achieved by the aforementioned contributions, these neural video codecs (in particular those based on autoencoder backbones) are typically associated with high computational complexity (especially in the decoder), which constrains their deployment for practical applications. To address this issue, researchers are now focused on complexity reduction while maintaining coding performance through model pruning and knowledge distillation (Guo-Hua et al. 2023; Peng et al. 2024).

It should be noted that the neural codecs mentioned above are typically trained offline with diverse video content (Nawała et al. 2024), and deployed online for inference. In this case, model generalization becomes important, and this is why these codecs often have a large model capacity, resulting in large model sizes and slow inference runtime. Inspired by recent advances in implicit neural representations (INR), a new type of video codec has emerged that employs INR models to “represent” the video by learning a coordinate-based mapping and compressing the network parameters for transmission. This approach converts a video coding problem into a model compression task, which allows the use of a much smaller network to “overfit” the input video, with the real potential for fast decoding. Existing implicit neural video representation (NeRV) models can be classified into index-based and content-based methods. The former takes frame (Chen et al. 2021), patch (Bai et al. 2023) or disentangled spatial/grid coordinates (Li et al. 2022) as model input, while content-based approaches (Kwan et al. 2024; Kim et al. 2024; Leguay et al. 2024) have content-specific embedding as inputs. Currently, one of the best INR-based video codecs (Kwan et al. 2024) has already achieved a performance similar to that of VVC VTM (RA), but with a much lower decoding complexity compared to autoencoder-based neural codecs. Some of these models have also been applied to volumetric video content (Ruan et al. 2024; Kwan et al. 2024), demonstrating their potential to compete with standard and other learning-based methods. However, it should be noted that the training of most NeRV models is based on an entire video sequence or even datasets; this results in a high system delay and does not meet the requirement of many low latency video streaming or real-time applications. To address this limitation, significant advances have been made (Gao et al. 2024) towards more practical INR-based video compression (such as the Low Delay and Random Access modes in VVC VTM (Bossen et al. 2023)) by combining pre-training and online model overfitting.

Similarly to image compression, international grand challenges are used to compare neural video compression methods, with notable venues including the NN-based Video Coding Grand Challenge associated with The IEEE International Symposium on Circuits and Systems (ISCAS) and the Challenge on Learned Image Compression (CLIC, video coding track) with IEEE/CVF CVPR and Data Compression Conference (in 2024). The best performer in ISCAS 2024 NN-based Video Coding Grand Challenge offers an overall 55% BD-rate saving over HEVC Test Model HM (ISCAS 2024), while the winner of the CLIC (video coding track) in 2024, a neural-network enhanced ECM codec (Zhao et al. 2024) with a CNN-based in-loop filter, shows a more than 2dB (in PSNR) gain compared to VTM (RA) at the same bitrates.

Audio compression

Similarly to images and videos, learning-based solutions have also been researched to compress audio signals, and most neural audio codecs are based on VQ-VAE (Oord et al. 2017). SoundStream (Zeghidour et al. 2021) is one such model, which can encode audio content at various bitrates. It is based on a residual vector quantizer (RVQ) which trades off rate, distortion and complexity. This work has been further enhanced with a multi-scale spectrogram adversary and a loss balancer mechanism, resulting in improved rate-distortion performance. A more advanced universal model has been further developed (Kumar et al. 2023) based on improved adversarial and reconstruction losses, which can compress different types of audio. RVQ has also been extended from a single scale to multiple scales (Siuzdak et al. 2024), which performs hierarchical quantization at variable frame rates.

More recently, researchers have started to exploit the use of LLMs for audio compression, leveraging the audio generation/synthesis abilities of generative models. UniAudio 1.5 (Yang et al. 2024) is one of such attempts, which converts an audio into the textural space, which can be represented by a pre-trained LLM that shares a similar backbone of UniAudio (Yang et al. 2023), a universal audio foundation model. LFSC is another neural audio codec based on LLMs, which achieved fast LLM training and inference through finite scalar quantization and adversarial training.

Visual quality assessment

Assessing the quality of visual signals remains an important and challenging task for many image and video processing applications. While subjective tests involving human participants remain the gold standard, objective quality models are frequently used because of their time and cost efficiency. These quality assessment methods are typically used to evaluate the performance of different visual processing approaches, and they can also be converted to loss functions, which are employed for optimizing learning-based processing models.

In recent years, quality assessment methods have been enhanced using deep learning techniques. The resulting learning-based quality models can quickly adapt to a specific type of content, leading to better performance compared to conventional, hand-crafted quality metrics. This section provides a brief summary of existing work in this research area, and highlights the main challenges which should be addressed in the near future. A more comprehensive overview of the image and video quality assessment literature can be found in (Zhai and Min 2020; Zheng et al. 2024; Zhang et al. 2024).

Quality assessment models

Image and video quality assessment methods can be classified into two primary categories according to the availability of the corresponding reference (un processed) content. These are referred to as full-reference and no-reference models.55 Prior to the AI era, conventional visual quality methods often exploited characteristics of the human vision system capturing information related to structural similarity (such as in SSIM and its variants (Wang et al. 2004, 2003; Rehman et al. 2015)), distortion (Chandler and Hemami 2007; Larson and Chandler 2010; Vu et al. 2011), and artifacts (Ou et al. 2010; Zhu et al. 2014; Zhang and Bull 2015). In many cases, the extracted features are further processed by models that simulate texture masking (Helmholtz 1896), contrast sensitivity (Kelly 1977), and saliency (Itti and Koch 2001). These hand-crafted quality models have also been combined with features within a regression-based framework in order to achieve more accurate prediction performance - VMAF is one such example (Li et al. 2018). When neural networks are used for feature extraction, they are trained to capture information which can directly contribute to quality prediction through an end-to-end optimization strategy. Initially, convolutional neural networks were used for this, with notable examples such as DeepQA (Kim and Lee 2017), LPIPS (Zhang et al. 2018) and CONTRIQUE (Madhusudana et al. 2022) for image quality assessment, and TLVQA (Korhonen 2019), C3DVQA (Xu et al. 2020) and DeepVQA (Kim et al. 2018) for video quality assessment. Recent works have been reported to achieve better performance when Vision Transformers (ViTs) (or similar variants) are employed due to the effectiveness of their self-attention mechanism. Important works in this class include IQT (Cheon et al. 2021), TRes (Golestaneh et al. 2022), SaTQA (Shi et al. 2024), FastVQA (Wu et al. 2022) and RankDVQA (Feng et al. 2024). The former has been further extended as DOVER (Wu et al. 2023) and COVER (He et al. 2024) when aesthetic and/or semantic aspects in the content are taken into account.

More recently, inspired by the success of large language models (LLMs) (Open et al. 2023; Touvron et al. 2023) in other machine learning tasks, these have been utilized in image and video quality assessment, demonstrating significant potential to achieve better model generalization. Q-Bench (Wu et al. 2024) is one of the first attempts that employs multimodal large language models to predict the perceptual quality of images based on prompt-driven evaluation. It queries the LLMs to provide information related to the final quality rating of the input image and the quality description. This has been further extended for video quality assessment tasks in Q-Align (Wu et al. 2024). Other notable works include X-iqe (Chen et al. 2023) that performs the quality prompt in a multi-iteration manner focusing on both image fidelity and aesthetics. Prompt-based approaches have also been proposed for differentiating the quality difference between multiple images, such as 2AFC-LMMs (Zhu et al. 2024) based on a two-alternative forced choice prompt and MAP (maximum a posteriori) estimation. Moreover, recent research works also focus on using pre-trained vision-language models, such as CLIP (Radford et al. 2021), which align better image and text modalities. Important examples in this class for image quality assessment include ZEN-IQA (Miyata 2024), QA-CLIP (Pan et al. 2023) and PromptIQA (Chen et al. 2025). Similar works have also been proposed for video quality assessment, such as BVQI (Wu et al. 2023a, b) and COVER (He et al. 2024).

To support the training and validation of learning-based quality assessment models, image or video databases containing ground-truth subjective quality scores are typically employed. Commonly used image quality databases include LIVE (Sheikh et al. 2006), CSIQ (Larson and Chandler 2010), TID2013 (Ponomarenko et al. 2013), PieAPP and PIPAL, while video quality databases such as LIVE-VQA (Seshadrinathan et al. 2010), KoNViD-1K (Hosu et al. 2017), YouTube UGC (Wang et al. 2019) and LIVE-VQC (Sinno and Bovik 2018) are typically employed for benchmarking in the literature. There are also databases developed that investigate the impact of specific video formats and/or artifacts, such as LIVE-YT-HFR (Madhusudana et al. 2021) focusing on frame rates, VSR-QAD (Zhou et al. 2024) on spatial resolution (or super-resolution artifacts), BAND-2k (Chen et al. 2024) on banding artifacts and Maxwell (Wu et al. 2023b)/BVI-Artifact (Feng et al. 2024) containing multiple artifacts commonly produced in video streaming. Based on these databases, many learning-based quality assessment models are trained to minimize the difference (L1 or L2 norm) between predicted quality indices and subjective scores. However, due to the limited number of ground-truth quality labels associated with these databases and the resource requirements associated with collecting subjective data using human participants in psychophysical experiments, this type of training methodology cannot offer satisfactory performance, in particular when the model capacity is large. Moreover, since the experimental settings and conditions used for quality labeling are different in these databases, intra-database cross-validation is always required due to the limited model generalization and potential overfitting problems.

To address these issues, various proxy quality metrics have been used to label images and videos, which avoid expensive subjective tests and enable the generation of a large amounts of training material with pseudo-ground-truth quality annotations. To further improve the reliability of quality labels, instead of learning the absolute values of the quality labels, ranking-inspired training strategies have been developed, which focus on improving the monotonicity characteristics of quality. Important examples based on these weakly supervised training methodologies include RankIQA (Liu et al. 2017) and UNIQUE (Zhang et al. 2021) for the image quality assessment task, and VFIPS (Hou et al. 2022) and RankDVQA (Feng et al. 2024) for video quality assessment. Moreover, different self-supervised learning approaches have also been employed, which transform quality labeling to an auxiliary task. For example, CONTRIQUE (Madhusudana et al. 2022) learns relevant features from an unannotated image database based on the prediction of distortion types and degrees through contrastive learning. This method has been further applied to video quality assessment, resulting in a contrastive video quality estimator, CONVIQT (Madhusudana et al. 2023). More recently, quality-aware contrastive loss has been designed in (Zhao et al. 2023; Peng et al. 2024) to stabilize the learning process.

Performance and main challenges

Due to the lack of standard test conditions and limited model generalization within many existing image and video quality assessment models, deep compression methods are typically trained and benchmarked using different databases in conjunction with intra-database cross-validation. This can result in inconsistent evaluation results and conclusions. To enable a fair and meaningful comparison, various challenges and contests have been held for visual quality assessment. The Sixth Challenge on Learned Image Compression (CLIC) (Challenge on Learned Image Compression 2024) associated with the Data Compression Conference 2024 is one of the latest examples which includes two quality assessment tracks for image and video compression. The best performer in the video quality assessment track achieves a Spearman Ranking Correlation Coefficient value of 0.825 (Feng et al. 2024), which is based on a ranking-inspired training methodology. Other notable challenges include the IEEE/CVF WACV 2023 HDR VQA Grand Challenge and the Video Super-Resolution Quality Assessment Challenge in ECCV 2024, which focus on high dynamic range and super-resolved content, respectively.

Although significant progress has been made in the past few years in visual quality assessment, including new models and training methodologies, challenges remain, including limited model generalization and high computational complexity. Another important use of quality metrics is as embedded loss functions for image and video processing optimization. This requires further capability and robustness, alongside complexity reduction, all topics to be addressed in future work.

Summary of AI technologies for creative industries

This section consolidates the preceding discussion by providing a comparative overview of the main classes of AI models shaping contemporary creative practice. While earlier sections examined individual technologies in detail, the following summary highlights how these models–ranging from large language models (LLMs) and diffusion models (DMs) to Neural Radiance Fields (NeRFs) and Implicit Neural Representations (INRs)–differ in application domains, key advantages, and persistent limitations. This synthesis enables readers from both creative and technical backgrounds to discern where each model type contributes most effectively, where challenges remain, and how these approaches collectively reshape workflows across the creative industries. Table 2 summarizes these relationships and serves as a conceptual reference for future research, mapping core model classes to their applications, strengths, and constraints to guide the evaluation and development of emerging AI methods in the creative industries.

Table 2.

Comparative summary of key AI technologies for creative industries

Technology	Core mechanism	Creative applications	Key strengths	Main limitations	Accessibility
Transformers/ attention	Self-attention mechanisms capture global dependencies	Image/video restoration, super-resolution, segmentation, object detection, tracking	Parallel processing, long-range context, scalable, state-of-the-art performance	High memory usage, quadratic complexity, requires large training datasets	High: SwinIR, SAM, Restormer (open-source)
Large language models (LLMs)	Language understanding and generation	Text generation, dialogue systems, screenwriting, storyboarding, code assistance, script analysis	Highly flexible across domains; capable of reasoning and context understanding; supports creative ideation and natural language interaction	Hallucinations and factual errors; limited interpretability; potential bias from training data; copyright and authorship ambiguity	Moderate: GPT-4, Claude (APIs); LLaMA, Qwen (open weights)
Diffusion models (DMs)	Iterative denoising from Gaussian noise	Image synthesis, Text-to-image/video generation, animation, VFX, design prototyping, restoration, style transfer, inpainting, restoration	High-quality diverse outputs; stable training; handles complex distributions; photorealistic; fine-grained control through prompts or conditioning inputs	Computationally expensive; slow sampling; temporal flicker in videos; difficulty ensuring semantic or style consistency	High: Stable Diffusion (open); DALLE 3, Sora (APIs)
Neural radiance fields (NeRFs)	Volumetric scene representation via MLPs	3D scene reconstruction, virtual production, spatial storytelling, AR/VR content creation	Photorealistic rendering from sparse views; compact scene representation; supports dynamic viewpoint changes	Limited to static or semi-static scenes; slow training; poor performance in textureless regions; large memory footprint	Moderate: Instant-NGP, Mip-NeRF (open); requires CV expertise
Implicit neural representations (INRs)	Coordinate-based continuous functions	Video/image compression; super-resolution; 3D reconstruction; dynamic scene modeling	Continuous scene representation; parameter-efficient encoding; smooth interpolation and compact storage	Challenging to generalize across scenes; training instability; limited semantic control	Moderate: NeRV, COIN (open); technical expertise required
Gaussian splatting	Explicit 3D Gaussians for scene representation	Real-time novel view synthesis; 3D scene reconstruction; VR/AR, interactive rendering	Fast training (minutes) and rendering (real-time); explicit representation; easy manipulation	Lower quality than NeRF for complex scenes; requires SfM initialization; memory intensive	High: 3D-GS, 4D-GS (open); LUMA AI, Polycam (free tools)

* Product and platform data (e.g., Sora, Gemini, Stable Diffusion 3) are accurate as of mid-2025 and may evolve rapidly

Closing thoughts: the future of AI in creative applications

This paper has presented a comprehensive review of current AI technologies and their creative industries applications that have emerged in recent years. Generative methods have driven a rapid growth in AI usage, particularly in the creative sector, significantly advancing the state of the art across various applications such as content creation, information extraction and analysis, content enhancement and data compression.

Through these applications, generative AI has not only broadened creative possibilities, but has also reduced the manual effort and time traditionally associated with the production pipeline, allowing for greater creative experimentation and more rapid and agile production cycles. As this technology advances, it promises to unlock even more sophisticated capabilities. However, creative technologists, artists and other users must adapt, learn to use, and build these tools effectively and safely.

Challenges for AI in the creative sector

Artists are already exploring how to bridge structured nature of current creative AI and more traditional (analog) workflows. This includes using multiple methods, models, or tools to create new works. For example, artists can use tools to iterate on existing or past works; upload and fuse analog works or output from different Generative AI tools to further intervene in the AI generation process; and composite outputs to reconstruct or fill in missing or damaged parts of a work. However, one of the primary challenges for artists engaging with modern generative AI and LLMs is the lack of consistent, controllable outputs. These models operate via stochastic sampling from high-dimensional latent spaces, meaning that identical prompts can yield different results across runs. This unpredictability can make it challenging for artists to achieve, or iterate toward, a precise creative vision. Although prompt engineering has emerged as a technique to guide model behavior, it requires technical knowledge and iterative refinement, which may not align with the intuitive or exploratory approaches common in artistic practice.

Moreover, there can be a fundamental tension between the structured nature of current AI pipelines and current production, often improvisational, workflows used in creative disciplines. Many generative tools were originally designed for tasks like software development, content automation, or optimization (Zhong et al. 2024), and are less well suited for open-ended, exploratory creation. Artists typically work in cycles of ideation, experimentation, and revision–processes that demand fluid, real-time interaction and control, which existing AI systems struggle to support. These limitations point to a gap in current AI design: a need for systems that not only generate high-quality content but also adapt to the iterative, interpretive nature of artistic production. One possible approach to address these challenges is a reinforcement of top-down creative workflows, where artists define high-level concepts, themes, or goals via text prompts before refining specific outputs. This approach helps align AI-generated results with artistic intent, offering a degree of control over inherently stochastic systems.

A further issue concerns creative authorship and ownership. As AI systems increasingly contribute to the ideation and execution of creative work, the line between human and machine authorship becomes blurred. Determining where creative credit lies–whether with the prompt designer, the model developer, or the AI system itself–poses significant legal and ethical challenges. Moreover, generative systems often reproduce stylistic elements from training data, raising questions about originality and cultural appropriation in AI-assisted creation.

Speaking at the World Government Summit in Dubai in 2024,56 NVIDIA CEO Jensen Huang argued that, with rapid advancements in AI, learning to code may become less essential for newcomers to the tech sector. He envisioned a future where traditional programming could be replaced by more intuitive AI-driven tools, thereby automating complex tasks and enhancing productivity–particularly for artists without coding expertise. While this perspective remains debated, it highlights the potential for AI to become more accessible within creative fields, not just in coding but across areas such as VFX and virtual production. But to achieve this, AI-assisted coding tools must be better integrated into creative workflows. Creators must exploit techniques such as fine-tuning pre-trained models, few-shot learning, or domain adaptation–methods that are powerful yet typically inaccessible without machine learning expertise.

There are also broader concerns that persist regarding the long-term impact of AI on the creative industries, economies, and labor markets. As automation accelerates, socioeconomic disparities may widen between those who can afford access to powerful generative systems and those who cannot. Freelancers and smaller studios risk being marginalized by large organizations with access to proprietary datasets and substantial compute resources. The potential emergence of artificial general intelligence (AGI). Envisioned by organizations like OpenAI, DeepMind, and Anthropic, AGI could surpass human cognitive abilities, raising ethical and existential questions about the role of human agency in artistic expression. Ensuring equitable access to AI tools and fair distribution of creative value will therefore be crucial to sustaining diversity, inclusion, and innovation within the sector.

Ethical issues, fakes and bias

Advancements in generative AI, exemplified by models like Sora and Gemini 1.5 Pro, provoke ethical concerns and have societal implications. While their applications, with appropriate permission, can be beneficial and entertaining, these models, because they are capable of generating highly realistic content, escalate the risk of misuse through malicious deepfakes and misinformation. We are now in a situation where AI results transcend the uncanny valley, further complicating matters and challenging perceptions of authenticity. For example, the artist Miles Astray demonstrated that even authentic photographs could be mistaken for AI-generated images. His real photograph ‘F L A M I N G O N E’ won both the jury’s award and the people’s choice award in the AI category of the 1839 Awards. His aim was to highlight the ethical dilemmas inherent in AI, suggesting that the benefits of discussing AI’s ethical implications could surpass the ethical concerns related to viewer deception.57

While democratizing AI tools no doubt presents opportunities to transform creative processes and workflows, it also necessitates robust regulatory frameworks to safeguard privacy and ownership. For example, deepfake technologies stimulate significant concerns about the spread of misinformation and other malicious uses. Efforts to detect and identify increasingly realistic deepfakes are thus as important as the generative methods used to produce them. These must however be accompanied by increased media literacy, and policies that address the ethical and legal implications.

Diversity and representation is a key issue when using AI tools. Unified Concept Editing (Gandikota et al. 2024) has been proposed as a basis for image generation in digital mediums. This aims to ensure the production of safe content with diverse representation, reducing gender and racial biases. Hallucination in generative AI (the production of outputs that are not faithful representations of reality but instead contain imagined or unrealistic elements) is a further cause of concern. These undermine trust in AI processes and can be due to limitations in the training data, biases in the model architecture or imperfections in the optimization process. Hallucinations associated with LLMs are one of the issues highlighted by the UK Government (Communications and Digital Committee 2024), alongside bias, regurgitation of private data, difficulties with multi-step tasks and challenges in interpreting black-box processes.

Governments across the world are increasingly expressing concerns about the challenges and uncertainties that generative AI technologies pose to rights holders and human creativity (Jeary and Gajjar 2024). Generative AI presents substantial legal challenges, including the copyright status of AI-generated work and the intellectual property and copyright implications of the datasets used in training AI models. Viewpoints on this issue do however differ. For example, the track “Heart on My Sleeve," penned by an (as yet unidentified) human author, featured AI-generated vocals that replicated the voices of Drake and The Weeknd. Released independently on April 4, 2023, it was accessible via streaming platforms including Apple Music, Spotify, and YouTube. The song quickly became viral, accumulating over 20 million views across all platforms,58 prior to its removal by Universal Music Group, Drake’s recording label. In contrast, Canadian artist Grimes has extended an invitation to musicians to emulate her voice via AI for the creation of new musical pieces, stipulating that the lyrics should not be harmful. She has advocated for the democratization of art and the abolition of copyright.59 Additionally, Grimes has employed AI to design visual content for her LED backdrop at Coachella in 2024.

Finally, the rapid development of AI technologies has also raised concerns about job displacement and the balance between automation and human participation in creative processes. Ensuring that AI augments, rather than undermines, human effort poses a significant challenge for developers and policymakers.

The future of AI technologies

Several key technological issues remain which need to be addressed if AI is to deliver its full potential. These in particular relate to training data, computational complexity and their depth of reasoning or planning, and are discussed below.

A substantial amount of data is essential for training AI models in order to achieve high performance and good generalization. Major companies such as Google, Meta, and NVIDIA, with their respective models: BERT, Segment Anything, and Canvas, dominate this space, benefiting from leveraged resources to gather data and process it to train sophisticated models. However, in November 2024, Bloomberg reported that OpenAI, Anthropic, and Google are all experiencing relatively slow growth in the performance of their AI models, with one of the key challenges being training data.60

LLMs excel in applications involving complex tasks, advanced reasoning, data analysis, and understanding context. However, these models typically require high computational resources or cloud computing for development, operation and fine-tuning. A new trend emerging alongside LLMs is the development of Small language models (SLMs), such as Phi-3 by Microsoft.61 SLMs offer promising solutions for regulated industries and sectors encountering scenarios where high-quality results are essential while keeping data ’on-site’. Their potential is particularly relevant when deploying more capable SLMs on smartphones and other mobile devices, allowing them to operate ‘at the edge’ without relying on cloud connectivity. Recent highly successful platforms, such as DeepSeek-V3 (DeepSeek et al. 2024) and Qwen2.5-Max (Team 2024), are based on Mixture-of-Experts (MoE) models, which tackle complex problems by dividing them into simpler sub-tasks, each handled by a specialized “expert.”

Despite evident advancements in AI, current models still struggle with tasks requiring planning or deep reasoning and are prone to errors when encountering unexpected data. This, in turn, reduces the confidence of users and trust in the results. AI algorithms can learn through reinforcement learning, but this process often identifies the best outcome as an anomaly rather than the norm. Yann LeCun, Professor at NYU and Chief AI Scientist at Meta, noted that while LLMs show a degree of comprehension in processing and generating text, their understanding lacks depth, often leading to results that defy common sense.62 He advocates for self-supervised learning as a pivotal future direction for AI, emphasizing its potential to derive insights from unlabeled data. Concurrently, Andrew Ng, Adjunct Professor at Stanford University and Founder of DeepLearning.AI, sees iterative AI agentic workflows63 as a key advancement for enhancing AI tool capabilities through an interactive approach by AI agents. These workflows involve autonomous agents that interactively learn from experience, understand natural language, and execute tasks on behalf of users.

The increasing openness of code and datasets is seen by many as a catalyst for accelerating AI advancements, with major firms like Microsoft, Google, and Meta supporting open access technologies. However, this openness also introduces security risks, necessitating new regulatory measures to monitor models post-release, to standardize documentation, and to assess the safety of software code and training data disclosure.

Finally, as stated in Jeary and Gajjar (2024), the rapid advancement of AI technologies has revolutionized cultural experiences, often referred to as ‘CreaTech’–the convergence of the creative and digital sectors (Council 2021). Such innovations not only reshape how people engage with art and creative work (e.g., through AR/VR/MR) but also drive the evolution of the technologies themselves.

Author contributions

NA wrote the main manuscript text in section 1, 2, 3.13.5, 3.8, 4, prepared all figures. FZ wrote the main manuscript text in section 3.6, 3.7. DB wrote the main manuscript text in section 4. All authors reviewed the manuscript.

Funding

This work has been funded by the UKRI MyWorld Strength in Places Programme (SIPF00006/1).

Data availability

No datasets were generated or analysed during the current study.

Declarations

Conflict of interest

The authors declare no conflict of interest.