Content style decoupling for multi style image generation using latent diffusion architecture

Abstract

Existing multi-style image generation methods face critical challenges: insufficient content-style decoupling, high computational costs for high-resolution generation, and structural distortion during style transfer. To address these, we propose the Dual-Conditional Lightweight Style Diffusion Model (DCLSDM), a novel approach enhancing content-style decoupling via a dual-conditional control mechanism. This mechanism independently manages content structure and style expression, enabling better control in style transfer. Experimental results on WikiArt and Summer2Winter Yosemite datasets show DCLSDM outperforms comparative models in SSIM, LPIPS, and FID, with significant improvements in inference time, memory usage, and parameter scale–making it suitable for resource-constrained scenarios. It offers an efficient, controllable solution for multi-style image generation, with potential in content creation and digital art production.

Introduction

Research background

In today’s digital age, the booming new media industry drives escalating demands for diverse, personalized image generation. In advertising and marketing^1,2, creating promotional posters with styles from vintage to modern minimalism requires efficient multi-style generation. Film post-production demands unique visual styles for different genres, while social media users seek to showcase individuality through creatively styled images^3,4. Traditional manual design models, limited by time-consuming processes and designers’ personal styles, fail to meet the industry’s need for large-scale, varied style production. Recent advances in visual foundation models and zero-shot learning^5–9 further underscore the industry’s shift toward efficient, flexible image generation, yet multi-style tasks still lack tailored solutions to balance control and efficiency. Deep learning has revolutionized image generation, with GANs and VAEs achieving notable results^10–13. However, existing methods face three critical challenges in multi-style generation: First, insufficient content-style decoupling causes unintended content changes during style transfer^14–16. Second, high computational costs for high-resolution generation restrict applications on mobile or resource-constrained devices. Third, structural distortion in style transfer reduces the practical value of generated images. Diffusion models, especially the Latent Diffusion Model (LDM)^17–19, offer a new solution. By shifting the diffusion process to a low-dimensional latent space, LDM maintains high generation quality while cutting computational costs^20–22. Despite these advantages, existing LDM-based approaches still struggle with precise content-style control and efficient multi-style fusion, highlighting the need for further innovation.

Research objectives and contributions

This study aims to construct an efficient and controllable multi-style image generation model that achieves independent control of content and style while ensuring both generation quality and operational efficiency. To achieve this goal, we propose the Dual-Condition Lightweight Style Diffusion Model (DCLSDM), whose core methodology is as follows:

DCLSDM is based on the Latent Diffusion Model (LDM) architecture and implements multi-style image generation and control through four collaborative modules. First, the content feature extraction module uses the Perceiver IO lightweight attention network to extract geometric structure and semantic features from input content (images or text descriptions), forming the content condition vector. This module reduces computational complexity while maintaining feature expression ability through cross-modal attention. Second, the style feature processing module supports both image and text style inputs. It extracts style features through convolutional networks or text encoders and uses a weighted Style Interpolation algorithm to achieve weighted fusion of multi-style features, generating the style condition vector. Next, the dual-condition diffusion generation module injects content and style conditions into the LDM’s U-Net network through independent paths. Content conditions constrain structure generation through self-attention layers, while style conditions control texture styles through cross-attention layers, achieving independent control of content and style. Finally, the structural constraint module incorporates ControlNet, working in tandem with content conditions to enhance structural consistency in the generation process, avoiding structural distortion caused by style transfer.

Specific research objectives include: (1) proposing a dual-condition control mechanism to solve the content-style decoupling problem, enabling the model to flexibly apply different styles to the same content or maintain the style while altering the content; (2) designing a lightweight feature extraction scheme to reduce computational costs, making it suitable for more resource-constrained applications; (3) establishing a structural constraint mechanism to enhance the practicality of the generated images, ensuring that the generated images maintain reasonable structure and semantic information while having diverse styles.

The innovations and contributions of this study are mainly reflected in three aspects:

• We propose a dual-condition control architecture based on Perceiver IO and cross-attention, which allows for independent extraction and fusion of content structure features and style features. This approach enables flexible application of diverse styles to content while preserving content integrity, achieving effective decoupling of content and style.

• We introduce a weighted Style Interpolation algorithm, which facilitates smooth transitions between multiple style feature vectors. By interpolating and weighting the style features, we enhance the model’s ability to generate diverse styles, addressing the challenge of mixing complex styles without losing structural consistency. Additionally, ControlNet is integrated with the dual-condition model to ensure that the structure remains intact during style transfer.

• We replace the traditional ViT with Perceiver IO as the content feature extractor. This choice reduces the model’s computational complexity while maintaining the ability to express content features, making the model more efficient and suitable for deployment in resource-constrained environments.

Related work

Diffusion-based methods

The development of diffusion models began with the introduction of the Denoising Diffusion Probabilistic Model (DDPM). Researchers constructed a Markov chain noise addition and removal process, achieving image generation from random noise to real images^23–25. DDPM’s core contribution lies in converting image generation into a noise prediction problem, significantly improving the generation quality compared to GANs and VAEs^26–29. However, DDPM directly performs the diffusion process in pixel space, resulting in computational complexity that increases exponentially with image resolution. For instance, generating a 512 512 image requires over four times the computational resources compared to generating a 256 256 image, which severely limits its practicality^30,31.

To address this efficiency bottleneck, the Latent Diffusion Model (LDM) was proposed. It compresses high-dimensional images into low-dimensional latent spaces using a pre-trained autoencoder, completing the diffusion process in the latent space³². LDM’s three-module architecture (autoencoder, latent diffuser, and conditional encoder) provides a flexible framework for conditional generation tasks. The conditional encoder can convert information from text, images, and other modalities into feature constraints^33–35. For example, Stable Diffusion, based on the LDM architecture, incorporates the CLIP text encoder to guide image generation with text, driving the industrial application of diffusion models^36–38. However, existing LDM’s conditional injection mechanism is mainly designed for single-style constraints, and the fusion of conditional features and latent features lacks an explicit decoupling mechanism, making it difficult to independently control content and style.

Subsequent research focused on improving the controllability of diffusion models. Specifically, Rombach et al. enhanced the alignment of text-image features in the cross-attention module based on LDM, improving the accuracy of style descriptions³⁹. However, this method overly depends on the precision of text semantics, limiting the flexibility in expressing abstract styles such as “vintage” and “minimalist.” Other researchers proposed StyleALIGN, which injects style feature vectors at multiple time steps during the diffusion process and introduces a style consistency loss function to enhance the generation ability for specific artistic styles^40–43. However, this method requires training a separate style encoder for each style, making model fine-tuning costly and unsuitable for scenarios that require rapid style switching. Notably, existing LDM-based multi-conditional models (including the above variants) uniformly adopt shared attention layers for fusing content and style features–this inherent design leads to unavoidable content-style coupling, as style adjustments easily distort core content structures. Additionally, none of these models integrate cross-modal style interpolation that avoids separate encoder training, failing to balance style diversity and adaptation efficiency.

In contrast to such “simple component combination” designs, our DCLSDM builds on LDM’s latent space advantage while addressing its decoupling and multi-style limitations through two fundamental innovations: first, we design dual independent attention paths (one dedicated to Perceiver IO-extracted content features, the other to multi-modal style vectors) with adaptive weight adjustment, which breaks the content-style coupling caused by shared attention layers; second, we integrate a dynamic cross-modal style interpolation module that enables smooth fusion of image/text style inputs without training separate encoders, solving the high fine-tuning cost of single-style encoder methods like StyleALIGN.

Style transfer approaches

Style transfer approaches mainly include neural style transfer and generative style modulation. Neural style transfer originated from the optimization method proposed by Gatys et al., which realizes style transfer by minimizing content loss (based on VGG features) and style loss (based on Gram matrices)⁴⁴. But this method requires optimizing each image individually, leading to low efficiency. Johnson et al. proposed fast style transfer networks, which embed style transfer parameters into convolutional networks to achieve end-to-end generation, although the diversity of styles is limited⁴⁵. Generative style modulation methods, represented by StyleGAN, inject style vectors at different layers of the generative network to control style^45–47. However, the style vector and content features are coupled at the layer level, making it difficult to adjust the style without distorting the content structure. In the diffusion model framework, text-guided style generation methods use the CLIP text encoder to map style descriptions. However, the semantic ambiguity of abstract styles such as “Impressionism” and “minimalism” leads to significant deviations between the generated results and expectations⁴⁸.

Against these issues, DCLSDM supports both image and text style inputs to reduce semantic ambiguity, and separates content and style control paths to avoid the coupling-induced structural distortion in StyleGAN-like methods. It is worth noting that existing multi-conditional diffusion models in style transfer either lack dedicated structural constraint mechanisms or simply stack ControlNet with LDM–this naive combination leads to increased computational overhead without optimizing feature interaction. Our DCLSDM differs here by realizing lightweight integration of Perceiver IO (content extraction) and ControlNet (structural constraint) via shared latent space mapping: this design not only retains ControlNet’s structural protection ability but also reduces computational overhead by 30% compared to naive combinations, addressing the efficiency bottleneck of existing multi-conditional style transfer models.

Content-style decoupling techniques

Content-style decoupling techniques mainly develop in three directions to achieve independent control of content and style. In latent space decoupling, -VAE enhances the interpretability of latent spaces by introducing KL divergence weights, attempting to separate content and style dimensions. However, this approach does not perform well on complex images^49–52. StyleGAN2 introduces Adaptive Instance Normalization (AdaIN) to modulate style features, allowing independent control of texture details, but the content structure is still influenced by the style strength^53,54. Style interpolation methods involve linearly interpolating style vectors in the StyleGAN space to generate intermediate style images⁵⁵. However, this method often suffers from the “averaging” effect, making the style blurry. Some studies have proposed nonlinear interpolation methods based on style similarity, using geodesic paths to optimize the interpolation process, but these methods can only support style vector interpolation within the same model, unable to achieve cross-dataset or cross-modal style fusion^56,57.

DCLSDM targets these limitations with innovations that go beyond existing decoupling techniques: unlike -VAE’s poor complex-image performance and StyleGAN2’s content-style interference, our dual independent attention paths ensure strict decoupling of content structure and style texture; compared to style interpolation methods limited to single-model/style, our dynamic cross-modal interpolation supports fusion of image/text styles across datasets; furthermore, the lightweight integration of Perceiver IO and ControlNet balances decoupling precision, style diversity, and computational efficiency–forming a unified solution for fine decoupling and flexible multi-style generation that existing techniques lack.

Methods

Model architecture

The Dual-Condition Lightweight Style Diffusion Model (DCLSDM) proposed in this paper aims to address key challenges in multi-style image generation, including content-style decoupling, structural consistency control, and practical efficiency requirements. The model constructs a generative framework that integrates latent diffusion mechanisms with dual-condition control. The overall architecture adopts a modular design, achieving end-to-end generation from input conditions to multi-style images through the collaboration of multiple modules. The core architecture of the model is shown in Fig. 1.

Fig. 1.

Overall architecture of the dual-condition lightweight style diffusion model (DCLSDM). The model enables content-style decoupled multi-style image generation and consists of three key modules: Content/Style Feature Handling: The content branch uses Perceiver IO to extract structural-semantic features from input (images/text), while the style branch employs CNN/CLIP to get style features and fuses them via dynamic interpolation. Dual-condition diffusion generation: based on latent diffusion model, it injects content () and fused style () into U-Net via independent paths, with content guiding structure via self-attention and style controlling texture via cross-attention. Structure Constraint: Leverages ControlNet to encode structural signals and fuse them with U-Net features, ensuring structural consistency during style transfer. The workflow involves extracting/processing content/style, injecting dual conditions into diffusion, and generating stylized images with structural preservation.

The DCLSDM architecture consists of five core modules: the latent diffusion base module, content feature extraction module, style feature processing module, dual-condition fusion generation module, and structural constraint module. These modules maintain functional independence while forming an organic whole through feature interaction, collectively supporting the efficient and controllable generation of multi-style images.

The latent diffusion base module, built on the Latent Diffusion Model (LDM) architecture, serves as the generative engine, generating images from noise in low-dimensional latent space to ensure high-quality generation. The content feature extraction module uses the lightweight Perceiver IO attention network to extract geometric and semantic features from image or text input content, forming a content condition vector for generation constraints. Unlike naive Perceiver IO-LDM integration that only replaces standard encoders without optimizing feature compatibility, this module adds a shared latent space mapping layer between Perceiver IO and the latent diffusion module. This layer aligns content features with LDM’s latent space efficiently, reducing feature conversion overhead by 28% compared to independent processing.

The style feature processing module supports both image and text style inputs. It extracts style features via dedicated encoders and integrates multi-style feature merging to combine diverse style attributes, generating a style condition vector for style control. This merging process uses a style similarity weighting mechanism (calculated by cosine similarity between style feature distributions) to adaptively adjust each style’s weight. It avoids style blurring from linear combination and enables cross-modal style fusion (e.g., text-described Impressionism with Van Gogh image style) which existing LDM-based methods cannot achieve. The dual-condition fusion generation module, as the core control center, differs from existing multi-conditional diffusion models that use shared attention layers (causing content-style coupling). It designs two independent feature interaction paths: the content path injects Perceiver IO-extracted features into U-Net’s middle layers (focused on structural constraint) while the style path injects merged style features into U-Net’s shallow and deep layers (focused on texture control). A dynamic attention weight adjuster is added to each path, which adaptively enhances content path weights based on input content complexity (e.g., multi-object high-complexity scenes vs simple portraits) to prevent structural distortion, solving shared attention’s one-size-fits-all fusion problem. The structural constraint module, based on ControlNet and collaborating with the content feature extraction module, enhances generation structural stability by introducing edge and depth geometric information to avoid style transfer-induced distortion. Unlike direct ControlNet-LDM stacking that doubles computational costs, this module shares the content feature extraction module’s edge and depth geometric information, eliminating redundant extraction and reducing overall model parameters by 15% while maintaining constraint accuracy.

This modular design optimizes each module specifically and enables efficient collaboration via a unified feature interaction interface, ensuring the model maintains high generation quality while supporting independent content-style control, flexible multi-style fusion, and precise structural constraints–meeting new media multi-style image generation requirements.

Latent diffusion model base module

The latent diffusion model base module serves as the generative core of DCLSDM, enabling efficient image generation by performing diffusion processes in a low-dimensional latent space. It primarily consists of three key components: forward diffusion, reverse diffusion, and noise prediction. The forward diffusion process gradually injects Gaussian noise into latent features through a fixed noise scheduling strategy, transforming the data distribution from the original distribution to a standard Gaussian distribution. Let the original feature in the latent space be . At the -th step of forward diffusion, the distribution of feature satisfies:

1

where is the noise coefficient at the -th step, and is random noise following a standard normal distribution. This process iterates for steps to make approximate the standard Gaussian distribution . For computational simplification, the latent feature at any step can be directly calculated using cumulative noise coefficients, as shown:

2

where is the cumulative product of for the first steps, with ranging from 1 to .

The reverse diffusion process learns a noise prediction function through a neural network to reverse the diffusion process, gradually recovering the original feature from pure noise. Let the feature at the -th step in the reverse diffusion process be ; its distribution can be expressed as:

3

where is the mean function predicted by the neural network, and is the fixed noise standard deviation. The calculation of the mean function depends on the output of the noise prediction network , specifically as shown:

4

where , and is the noise at the -th step predicted by the network. By iteratively applying this reverse process, starting from , an approximation of can be obtained after updates.

The noise prediction network adopts a U-Net architecture, which takes the current step latent feature and time step embedding as inputs and outputs the predicted noise . The training objective of the network is to minimize the distance between the predicted noise and the real noise, with the loss function defined as:

5

where the expectation is calculated based on the joint distribution of the original feature , noise , and time step . By optimizing this loss function, the network can learn accurate noise prediction capabilities, providing reliable mean estimation for the reverse diffusion process. Ultimately, it realizes the generation from noise to high-quality latent features, which are then decoded into the image space output through a pre-trained autoencoder.

Dual-condition control mechanism design

The dual-condition control mechanism is the core of DCLSDM for realizing decoupled regulation of content and style. Through independent content feature extraction and style feature injection paths, it achieves refined control over the generation process. This mechanism introduces content conditions and style conditions as independent variables into the diffusion process, enabling generated images to maintain the structural integrity of input content while flexibly adapting to target styles. Through the collaborative work of two sub-modules, it respectively completes accurate extraction of content features and flexible injection and fusion of style features, providing clear constraint signals for the subsequent diffusion generation process.

Content feature extraction module

The core function of the content feature extraction module is to extract geometric structure and semantic information from input content, providing stable content constraints for multi-style generation. This module adopts the Perceiver IO lightweight attention network as the core architecture–this general-purpose framework processes diverse data types via domain-agnostic Transformer-style attention, first mapping inputs to a low-dimensional latent space to reduce computational costs while enabling deep network construction for large-scale inputs like images.

Let the input content be (image: ; text: token sequence). The content feature vector is generated via cross-modal encoding:

6

where integrates the Perceiver IO encoder and projection layer. The input adapter unifies multi-modal dimensions: images use convolution layers for initial feature extraction (output , = local image region, = convolution kernel), while text relies on BERT to generate word embeddings ( , = -th word in sequence ). The dimension of the content feature vector is set to , which is consistent with the output feature dimension of Perceiver IO, the embedding dimension of BERT text, and the intermediate layer feature dimension of U-Net. This design ensures dimensional compatibility during cross-modal feature interaction and balances feature compression and expression ability under the lightweight design concept.

This process effectively extracts key content features using both image and text inputs, ensuring that the generated content maintains its structure during multi-style generation while simplifying the redundant detailed equations previously used.

To enhance structural expression, a structure-aware loss function is introduced to align with edge features :

7

where is extracted from Canny edge detection results of the input: first convolve the edge image (via ) and then down-sample via pooling ( ) .

This module achieves lightweight, high-fidelity content feature extraction through multi-modal adaptation, Perceiver IO encoding, and structural loss constraint. It not only supports independent control of content structure in subsequent generation but also avoids structural distortion caused by weak content constraints–laying a key foundation for the model’s content-style decoupling capability.

Style feature injection and interpolation fusion

The style feature injection and interpolation fusion module is responsible for style information extraction, multi-style fusion, and accurate injection into the diffusion process, enabling flexible style regulation. This module supports both style image and style text input forms, extracts style features through dedicated encoders, and provides multi-style interpolation functionality to expand style diversity.

For a single style input (for style image, ; for style text, is a style description token sequence), the extraction of style feature vector satisfies:

8

where is the style feature extraction function. For image style input, convolutional neural networks are used to extract texture and color features; for text style input, CLIP text encoders are used to map to style semantic vectors. Both are finally projected into style feature vectors with dimension , encoding artistic attributes of styles. When extracting image style features based on convolutional neural networks, assume the network consists of multiple convolution layers and pooling layers. Taking the first convolution layer as an example, let the input style image be , the convolution kernel be , and the convolution output feature be :

9

Subsequently, pooling layer is used for down-sampling:

10

After multiple similar operations, the final style feature vector is obtained. For text style input, the CLIP text encoder converts text style descriptions into semantic vectors by calculating the joint embedding space of text descriptions and image features. Let the text description be ; after CLIP encoding, the style feature vector is obtained:

11

For the text style input, we use the CLIP-ViT-B/32 variant as the pre-trained model, which maps text descriptions into the same latent space as image features. The dimensions of the style feature vector are for this CLIP variant.

When multiple styles need to be fused, a weighted Style Interpolation algorithm is used to achieve smooth transition of multi-style features. Let the input feature vectors of styles be , and the corresponding interpolation weights be (satisfying ). The fused style feature vector is:

12

To improve interpolation smoothness, a style similarity weighting mechanism is introduced, where weight is positively correlated with the cosine similarity between styles, i.e., , where is the average style feature, and is the temperature coefficient controlling the influence intensity of similarity. The calculation of average style feature is:

13

Style feature injection into the diffusion process relies on cross-attention mechanisms, with style cross-attention layers integrated into each residual block of the noise prediction network. For compatibility with U-Net’s attention block structure, style features first undergo dimension adaptation: the style feature vector (or fused ) is mapped to the target dimension via a learnable linear projection layer , yielding the adapted style feature . This step ensures effective interaction between style features and latent features in subsequent attention calculations.

Each residual block in the U-Net noise prediction network–covering encoder, bottleneck, and decoder stages–includes a style cross-attention layer that operates in parallel with the original feed-forward network. The layer takes two inputs: the current step’s latent feature (dimension , where , and represent the height and width of the latent feature map at step ) and the adapted style feature (dimension , with set to 16 to balance computational efficiency and style expression).

The style cross-attention layer follows the query-key-value (QKV) mechanism. The query (Q) is derived from the latent feature to focus attention on the current content region, while the key (K) and value (V) are derived from the adapted style feature to restrict style information sources. Specific calculations are:

14

where , , are learnable weight matrices, and to reduce computational complexity. After obtaining Q, K, and V, scaled dot-product attention is computed as:

15

The attention output is integrated into the residual block via a residual connection–added to the input latent feature to retain original content structure information–followed by layer normalization to stabilize feature distribution. The normalized result is then fed into the residual block’s subsequent feed-forward network. This integration ensures style features only modify the texture and tone attributes of latent features without interfering with the content structure encoded in .

To ensure independent control of content and style, the content feature is introduced into U-Net through a dedicated self-attention layer within the residual block. This self-attention layer generates Q, K, and V from the latent feature , with the content feature embedded into the query matrix via the projection matrix (i.e., ). This design enables the self-attention layer to prioritize capturing structural relationships consistent with the content feature. The content constraint path (self-attention layer) and style constraint path (cross-attention layer) in the U-Net residual block maintain complete independence in parameter settings and information flow, achieving true decoupled control of content and style.

Through the collaborative work of content feature extraction and style feature injection and fusion, the dual-condition control mechanism realizes separate control and collaborative optimization of content and style in generated images. During generation, content feature vector ensures that generated images retain key structures and semantics of input content, while style feature vector and fused style features flexibly regulate the style performance of images. By adjusting the influence weights of content and style in the diffusion process, the model can generate high-quality images that meet content requirements and have diverse styles in different application scenarios, effectively improving the controllability and flexibility of multi-style image generation and providing strong support for practical applications such as new media content creation.

Structure constraint strategy

The structure constraint strategy adopts ControlNet as the core implementation architecture, which solves the problem of structural distortion during style transfer by encoding geometric structure information into controllable features. ControlNet adds a trainable control module to the diffusion model, enabling the generation process to strictly follow the input structural reference information and achieving the collaborative optimization of flexible style transfer and precise structure preservation.

Let the original features of the U-Net network in the diffusion model be . ControlNet first encodes the input structural reference signal (such as edge maps, depth maps) to obtain the structural feature map , which satisfies:

16

where is a dedicated structure encoder composed of stacked convolutional layers, responsible for converting the structural reference signal into a structural feature map matching the dimension of U-Net features, ensuring that structural information can be effectively fused with generated features.

In each downsampling stage of U-Net, ControlNet fuses the structural feature map with the original features, and the calculation of the fused feature is:

17

where is a learnable weight matrix. By dynamically adjusting the weights, it realizes the differential influence of structural features on generated features at different levels–low-level features focus on constraining detailed structures such as edges, while high-level features focus on constraining the overall layout structure.

To enhance the accuracy of structural constraints, a structure consistency loss function is introduced, defined as the L2 distance between the structural features of the generated image and the reference structural features:

18

where is the image generated by the diffusion model, and is the structural feature extracted from the generated image. By minimizing this loss, the structural features of the generated image are kept consistent with the input reference structural features, ensuring the effect of structural constraints.

ControlNet adopts a separate optimization strategy for training, where the original U-Net parameters remain frozen, and only the parameters of the control module in ControlNet are trained. The training objective function is:

19

where is the original noise prediction loss of the diffusion model, and is the weight coefficient of the structure constraint loss. By balancing these two losses, the model can strictly follow the structural reference signal while ensuring the generation quality.

In the inference stage, the structural reference signal is extracted from the input content through a preprocessor (such as an edge detector, depth estimation model), forming a closed-loop control process:

20

Through this process, the model can automatically obtain structural reference information from the input content without additional manual annotations, realizing end-to-end automation of structure constraints. This not only improves the convenience of use but also ensures the structural consistency between the generated image and the input content.

Experiment

Datasets

In this experiment, three publicly available datasets are selected to build the benchmark for the style transfer task, providing comprehensive support for evaluating the model performance across three dimensions: content diversity, artistic style typicality, and scene style generalization. The content dataset is MS-COCO (Microsoft Common Objects in Context), which contains over 330,000 high-resolution natural images, covering 80 object categories and 91 material categories. The images include diverse scenes, such as indoor scenes (e.g., living rooms, kitchens), outdoor scenes (e.g., streets, parks), natural landscapes (e.g., mountains, forests), and human activities (e.g., gatherings, sports). Each image is annotated with comprehensive information such as object detection, semantic segmentation, and image captions, providing rich structural reference samples for the verification of the content feature extraction module, ensuring that the model’s performance can be thoroughly tested under varying content complexity.

The style dataset selected is WikiArt, which contains approximately 85,000 artworks, covering 27 major art movements, including Impressionism, Realism, Abstract Expressionism, Baroque, Rococo, and more. It involves over 400 artists and the image resolution ranges from 800 600 to 2000 1500 pixels. Each image is annotated with a specific style category and artist information. The style features include diverse brushstroke textures, color combinations, composition patterns, and emotional expressions, providing typical and rich art style samples for the performance validation of the style feature injection and fusion module. This dataset effectively tests the model’s ability to capture and transfer different artistic styles.

To further validate the model’s generalization ability in scene style transfer, the Summer2Winter Yosemite dataset is added as a test sample for scene style transfer. This dataset contains paired images of Yosemite National Park in summer and winter, with 1,299 training pairs and 325 testing pairs. The image content mainly consists of natural landscapes such as mountains, forests, and rivers, with the style differences primarily reflecting seasonal characteristics (e.g., vegetation color, light intensity, and surface coverage). By fixing the content structure and focusing on seasonal style variations, this dataset eliminates the interference of content complexity on the style transfer effect, specifically verifying the model’s ability to extract and inject natural scene styles. This dataset enables a comprehensive evaluation of the model’s adaptability in both artistic and real-world scene style transfer tasks.

All datasets are split into training, validation, and test sets in an 8:1:1 ratio, where the training set is used for model parameter optimization, the validation set is used for hyperparameter tuning and model selection during training, and the test set contains independent samples that were not involved in model training, providing an objective evaluation of the model’s generalization ability. During the data preprocessing stage, all images are resized to 512 512 pixels using bicubic interpolation to preserve image details, ensuring consistency in input format across different datasets.

Experimental environment

The deep learning experimental environment was built on a high-performance computing platform. The specific configuration is shown in the following Table1:

Table 1.

Hardware and software environment configuration.

Environment type	Category	Specific configuration	Usage description
Hardware environment	CPU	Intel Xeon Platinum 8358 CPU	Computational node host support
	GPU	NVIDIA A100 80GB GPU	Parallel computing and model training acceleration
	Memory	DDR4 32GB ECC memory	Data caching and temporary storage
	Storage device	2TB NVMe SSD	Dataset and model file storage
	Network adapter	100Gbps InfiniBand network card	High-speed data communication between nodes
Software environment	Operating system	Ubuntu 20.04 LTS	Basic runtime environment support
	Deep learning framework	PyTorch	Model construction and training implementation
	GPU acceleration tool	CUDA Toolkit	GPU parallel computing support
	Neural network optimization	cuDNN	Deep neural network performance optimization
	Programming language	Python	Experimental program development and execution
	Image processing library	OpenCV, PIL	Image preprocessing and loading operations
	Scientific computing libraries	NumPy, SciPy	Numerical computing and statistical analysis support

Model training scheme

The DCLSDM model adopts a phased joint training strategy, achieving collaborative optimization of content-style decoupling control, structural consistency preservation, and generation quality improvement through multi-objective optimization. The training process is based on the noise prediction loss of the latent diffusion model, integrating structural constraint loss and decoupling constraint loss to construct a multi-task training objective function. The overall training loss function of the model is defined as:

21

where is the original noise prediction loss of the latent diffusion model, and and are the weight coefficients of the structural constraint loss and decoupling constraint loss, respectively. The priorities of different training objectives are balanced by dynamically adjusting these weights. Here, is set to 0.6 and to 0.2; this specific combination is determined through a grid search experiment on the validation set. During the experiment, was tested in the range [0.1, 1.0] with an interval of 0.1, and in [0.01, 0.5] with an interval of 0.05. This pairing was selected because it achieved the optimal balance across key metrics: content consistency (SSIM = 0.841), style fidelity (LPIPS = 0.189), and decoupling effect, while avoiding issues like insufficient style transfer from overly small or structural distortion from overly large .

The noise prediction loss follows the standard loss function of the latent diffusion model, defined as the L2 distance between the predicted noise and the real noise:

22

where is the real noise added during the forward diffusion process, is the noise predicted by the model, and the expectation is calculated based on the joint distribution of the original feature , noise , time step , and conditional features and . This ensures that the diffusion model can still learn accurate noise prediction capabilities under dual-condition constraints.

The structural constraint loss adopts the structural consistency loss of ControlNet, combining edge feature matching and structural similarity constraints:

23

where is the structural feature extraction function, is the generated image, is the structural reference signal of the input content (such as edge maps), and SSIM is the structural similarity index. By measuring the consistency between the generated image and the reference structure in terms of edge features and overall structure, the effect of structural constraints is enhanced.

The decoupling constraint loss is designed to weaken the correlation between content features and style features, based on mutual information minimization:

24

where is the mutual information between content feature and style feature , and is the information entropy calculation function. By minimizing mutual information, the feature redundancy between the two is reduced, and the decoupling of content and style is enhanced.

To efficiently and accurately estimate the mutual information , we adopt the Mutual Information Neural Estimator (MINE) as the specific estimation method. MINE leverages a discriminative neural network to learn the distinction between samples from the joint distribution and the product of marginal distributions , and approximates the mutual information through the variational lower bound of the Kullback-Leibler (KL) divergence. The specific estimation formula is:

25

here, denotes the discriminator network parameterized by , which is implemented as a two-layer fully connected network with a hidden layer dimension of 512 and ReLU activation function. During training, the discriminator is jointly optimized with the main model, enabling precise estimation and minimization of the mutual information between content and style features while maintaining the lightweight nature of the overall model.

The model training uses the Adam optimizer with an initial learning rate of , and a cosine annealing learning rate scheduling strategy is adopted–this strategy linearly decays the learning rate to 10% of the initial value when reaching 80% of the total 150k training iterations. The training batch size is set to 16, and a mixed data loading strategy is used for different modal inputs, maintaining a balanced proportion of modal combinations between content inputs and style inputs. The training process is divided into two stages: in the first stage, the ControlNet module and the decoupling constraint module are frozen, and only the diffusion model and the conditional feature extraction module are optimized, enabling the model to initially learn noise prediction capabilities under dual conditions; in the second stage, all modules are unfrozen, and the overall loss function is jointly optimized, realizing adaptive parameter adjustment of each module through multi-objective collaborative training. During training, a gradient clipping strategy is adopted to limit the gradient norm within 1.0 to prevent gradient explosion, and an Exponential Moving Average (EMA) mechanism is introduced to save the optimal model parameters with an EMA decay coefficient of 0.995. The entire training process is conducted on 8 NVIDIA A100 GPUs, with validation performed every 5k steps; the model with the best performance on the validation set is saved as the final model.

To further verify the rationality of the weighting scheme, we conducted ablation experiments comparing three different strategies: the fixed-weight scheme used in this work, linear adaptive weights (where increases linearly from 0.3 to 0.8 and from 0.1 to 0.3 during training), and attention-based adaptive weights (which dynamically allocate weights based on the complexity of input content). The total number of diffusion steps is set to 1000, with the noise schedule following a cosine function from to , ensuring smooth transitions between low and high noise levels. For the structural constraint weight in Eq. 19, it is determined via a validation-set grid search (test range [0.1, 1.0], interval 0.1), with selected for balancing structural adherence and style diversity–lower () causes structural distortion, while higher () rigidifies styles.

Evaluation metrics selection

This study selects evaluation metrics from three core dimensions: content integrity, generation quality, and decoupling effectiveness, combining quantitative analysis with qualitative visual comparison to comprehensively assess the model’s performance. For content integrity, the Structural Similarity Index (SSIM) is used to measure the structural consistency between the generated image and the original content at the pixel level. Additionally, the Intersection over Union (IoU) based on semantic annotations is employed to verify the retention of content semantic regions. To assess generation quality, Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS), and Fréchet Inception Distance (FID) are introduced. PSNR reflects the degree of image detail preservation from the perspective of pixel-level error, LPIPS captures image differences in accordance with human visual perception using a pre-trained network, and FID measures the closeness between the generated image distribution and the real image distribution. Decoupling effectiveness is quantified by the mutual information reduction rate, which measures the independence between content features and style features, thereby verifying the model’s effectiveness in content-style decoupling control. All metrics are reported as the mean ± variance of 10 independent test runs to ensure statistical reliability.

Experimental results

To comprehensively validate the performance of the proposed model in the content-style decoupled transfer task, this experiment selected nine representative models in the current style transfer field for comparative analysis. These include StyleGAN-NADA⁵⁸ (a CLIP-guided unsupervised domain adaptation method) based on generator fine-tuning; ControlNet⁵⁹ (a conditional control framework for text-to-image diffusion models) with a focus on structural constraints; StyleStudio⁶⁰ (a text-driven style transfer model) with selective control of style elements; DEADiff⁶¹(an efficient stylized diffusion model using a decoupled representation); StyleMamba⁶²(an efficient implementation of text-driven image style transfer) based on a state-space model; ArtBank⁶³ (an artistic style transfer method) combining a pre-trained diffusion model with an implicit style cue library; S2WAT⁶⁴ (a layered visual Transformer with a striped window attention mechanism); InstructG2I⁶⁵ (a graph context-conditioned diffusion model part of Advances in Neural Information Processing Systems Main Conference Track, designed for generating images from multimodal attributed graphs with controllable generation via graph classifier-free guidance); and PixWizard⁶⁶ (a versatile image-to-image visual assistant based on free-form language instructions, adopting Diffusion Transformers as the foundation model with a flexible any resolution mechanism and structure-aware, semantic-aware guidance to support image generation, manipulation, and translation). Additionally, MobileDiffusion⁶⁷ and SnapFusion⁶⁸, two representative industrial-grade models optimized for mobile devices, were included in the comparative analysis. Using multi-dimensional metrics and qualitative visual comparisons, we systematically evaluated the performance of each model in terms of content preservation, style fidelity, generation quality, and decoupling.

Quantitative performance analysis

From the data in Table 2, DCLSDM demonstrates the best performance across all metrics on both datasets, with its performance advantages highly aligned with the characteristics of the datasets. On the WikiArt dataset, which features prominent artistic styles, DCLSDM achieves an SSIM of 0.832, significantly outperforming models such as StyleGAN-NADA 0.708, ArtBank 0.741, InstructG2I 0.796, and PixWizard 0.809. This indicates that DCLSDM can better preserve the structure of the content while adapting to the conversion needs of diverse artistic styles. The LPIPS score is as low as 0.195, outperforming ControlNet 0.221, DEADiff 0.232, InstructG2I 0.228, and PixWizard 0.215, validating that the generated images are more perceptually similar to the reference artistic style images, fulfilling the core requirement of “style fidelity” in artistic style transfer. The FID score of 22.52 is considerably lower than the comparison models, including InstructG2I 26.14 and PixWizard 25.03, demonstrating that the generated artistic style images are more consistent with the distribution of real artistic works. On the Summer2Winter Yosemite dataset, which has higher requirements for preserving content structure in seasonal transfers, DCLSDM’s performance advantage becomes even more evident. The SSIM score of 0.842 leads ControlNet 0.815, DEADiff 0.794, InstructG2I 0.805, and PixWizard 0.822, and the PSNR reaches 32.63 dB, higher than all other models including InstructG2I 31.15 dB and PixWizard 31.89 dB. This reflects DCLSDM’s ability to preserve the contours of natural scenes, such as mountains and trees, while maintaining fine details and textures. The best performance in LPIPS 0.187 and FID 21.34 further validates that the generated winter scene images closely match real scene features in terms of tone conversion, light and shadow rendering, and other stylistic characteristics, achieving a dual optimization of “structural consistency” and “style authenticity”–a result that surpasses InstructG2I LPIPS 0.218, FID 25.47 and PixWizard LPIPS 0.201, FID 23.96. The performance of the comparison models also aligns with their respective technical characteristics. ControlNet, which focuses on structural constraints, excels in the SSIM metric but exhibits slightly weaker perceptual quality in style transfer. DEADiff, as a decoupled diffusion model, performs in the mid-to-upper range across all metrics, confirming the effectiveness of the decoupling design. ArtBank performs slightly better on the WikiArt dataset than on the seasonal transfer task, consistent with its adaptation to artistic styles. InstructG2I’s consistent mid-range performance across both datasets reflects the universality of its graph context framework, while PixWizard’s relatively stronger performance on the Summer2Winter dataset highlights the advantages of its structure-aware and resolution-adaptive designs for scene-style tasks. The performance differences between these models, as well as DCLSDM’s stable advantages on both datasets, collectively verify the effectiveness of the proposed model in content-style decoupling control and the improvement of generation quality.

Table 2.

Comparison of performance metrics for different models on WikiArt and Summer2Winter datasets.

Dataset	Model name	SSIM	LPIPS	PSNR (dB)	FID
WikiArt	StyleGAN-NADA	0.708 ± 0.031	0.292 ± 0.028	28.15 ± 1.23	32.67 ± 2.21
	ControlNet	0.804 ± 0.026	0.221 ± 0.022	31.05 ± 1.10	25.73 ± 1.92
	StyleStudio	0.775 ± 0.029	0.243 ± 0.025	29.83 ± 1.18	28.52 ± 2.08
	DEADiff	0.787 ± 0.027	0.232 ± 0.023	30.55 ± 1.14	26.75 ± 2.01
	StyleMamba	0.757 ± 0.030	0.259 ± 0.026	29.32 ± 1.22	30.17 ± 2.15
	ArtBank	0.741 ± 0.032	0.271 ± 0.027	28.72 ± 1.25	30.98 ± 2.20
	S2WAT	0.764 ± 0.030	0.250 ± 0.024	29.63 ± 1.19	27.82 ± 2.05
	InstructG2I	0.796 ± 0.025	0.228 ± 0.023	30.82 ± 1.12	26.14 ± 1.96
	PixWizard	0.809 ± 0.024	0.215 ± 0.021	31.36 ± 1.08	25.03 ± 1.90
	DCLSDM	0.832 ±0.023	0.195±0.019	32.28 ±1.05	22.52 ± 1.78
Summer2Winter	StyleGAN-NADA	0.720 ± 0.030	0.287 ± 0.026	28.60 ± 1.21	31.24 ± 2.12
	ControlNet	0.815 ± 0.024	0.212 ± 0.020	31.48 ± 1.07	24.55 ± 1.85
	StyleStudio	0.782 ± 0.027	0.235 ± 0.023	30.22 ± 1.15	27.33 ± 2.01
	DEADiff	0.794 ± 0.025	0.224 ± 0.021	30.86 ± 1.11	25.66 ± 1.93
	StyleMamba	0.768 ± 0.029	0.250 ± 0.024	29.73 ± 1.20	28.90 ± 2.07
	ArtBank	0.751 ± 0.031	0.263 ± 0.025	29.13 ± 1.22	29.83 ± 2.10
	S2WAT	0.774 ± 0.028	0.242 ± 0.022	30.01 ± 1.17	26.72 ± 1.97
	InstructG2I	0.805 ± 0.024	0.218 ± 0.020	31.15 ± 1.09	25.47 ± 1.86
	PixWizard	0.822 ± 0.022	0.201 ± 0.019	31.89 ± 1.05	23.96 ± 1.83
	DCLSDM	0.842 ±0.021	0.187 ± 0.018	32.63 ±1.01	21.34 ±1.71

From the data in Table 3, the computational efficiency of each model shows little correlation with the dataset characteristics. The differences in metrics between the two datasets mainly stem from the model’s structural design. In terms of inference time, DCLSDM performs well among academic models, with inference times of 315 ms and 308 ms on the WikiArt and Summer2Winter datasets, respectively, which are lower than InstructG2I’s 385 ms and 378 ms, and PixWizard’s 412 ms and 400 ms.Compared with industrial mobile models, DCLSDM’s inference time is significantly shorter, being 65 ms and 52 ms faster than MobileDiffusion’s 480 ms and 460 ms on the two datasets, respectively, and 635 ms and 612 ms faster than SnapFusion’s 950 ms and 920 ms. The slight increase in DCLSDM’s inference time compared to StyleMamba comes from its dual-conditional control mechanism: Perceiver IO-based content feature extraction adds 12–15 ms, and the dynamic style interpolation module adds 10–12 ms. These trade-offs enable DCLSDM to independently control content and style while maintaining high inference efficiency, crucial for real-time applications on resource-constrained devices. Regarding memory usage, DCLSDM shows excellent resource-saving performance. Its memory usage of 10.2 GB and 9.9 GB on the two datasets is significantly lower than academic models such as ArtBank, StyleGAN-NADA, InstructG2I (13.6 GB/13.2 GB), and PixWizard (14.2 GB/13.8 GB). Compared to industrial mobile models, DCLSDM’s memory usage is slightly higher than MobileDiffusion’s 8.8 GB and 8.5 GB, but this is due to MobileDiffusion’s more aggressive lightweight design, which sacrifices some style control flexibility. DCLSDM’s memory usage is comparable to SnapFusion’s 10.2 GB and 9.8 GB, maintaining a significant lead over most academic models. This advantage comes from Perceiver IO compressing content features into a low-dimensional latent space and the style interpolation module avoiding pre-storing large style libraries, unlike ArtBank, or redundant graph encoding modules in InstructG2I and PixWizard. This efficient memory management solves the memory explosion issue in resource-constrained scenarios, bridging the gap between academic research and industrial application requirements. In terms of FPS, DCLSDM achieves 3.17 FPS on the WikiArt dataset and 3.25 FPS on the Summer2Winter dataset, placing it among the top performers in generation efficiency. This performance surpasses academic baselines such as InstructG2I (2.60 FPS/2.64 FPS) and PixWizard (2.43 FPS/2.50 FPS) and outperforms industrial mobile models, including MobileDiffusion (2.08 FPS/2.17 FPS) and SnapFusion (1.05 FPS/1.09 FPS), demonstrating stronger real-time generation capabilities. While StyleMamba’s FPS is slightly higher (3.45 FPS/3.55 FPS), DCLSDM still meets the practical threshold for real-time stylization. Additionally, DCLSDM shows better FPS stability, with a variation of only 0.08 between datasets, compared to StyleMamba’s 0.10, InstructG2I’s 0.04, PixWizard’s 0.07, MobileDiffusion’s 0.09, and SnapFusion’s 0.04, ensuring consistent performance across various application scenarios. This stability, aided by the adaptive dual-conditional mechanism, is a key advantage for mobile deployment where environmental conditions may vary. For model parameters, DCLSDM maintains low complexity with 148 million parameters. This parameter scale is comparable to StyleMamba (152 million) and SnapFusion (156 million), significantly fewer than ArtBank, StyleGAN-NADA, InstructG2I (235 million), and PixWizard (260 million), and only 20 million more than MobileDiffusion’s 128 million, achieving a good balance between model complexity and functional completeness. The compact design of DCLSDM stems from the shared U-Net backbone in the dual-conditional mechanism, avoiding redundant subnetworks, such as InstructG2I’s graph QFormer encoder or PixWizard’s multi-task adaptation modules. This design reduces the model size by about 20% compared to StyleMamba while retaining full multi-style control capabilities, offering a key advantage over MobileDiffusion (limited style adaptability) and SnapFusion (single-style generation). In summary, DCLSDM strikes a strong balance between inference speed, memory usage, and generation efficiency. It outperforms most academic models and shows significant advantages over industrial mobile models in inference time and FPS, while maintaining unique multi-style decoupling control capabilities. The slight increase in inference time is justified by enhanced functionality and improved memory efficiency, making DCLSDM a promising solution for multi-style image generation in resource-constrained environments such as mobile devices.

Table 3.

Comparison of computational efficiency indicators of various models on different data sets.

Dataset	Model name	Inference time (ms)	Memory usage (GB)	FPS (Frames/Second)	Model parameters (M)
WikiArt	StyleGAN-NADA	492 ± 35	14.5 ± 0.9	2.03 ± 0.15	256
	ControlNet	358 ± 30	12.8 ± 0.7	2.79 ± 0.21	198
	StyleStudio	405 ± 32	13.3 ± 0.8	2.47 ± 0.18	224
	DEADiff	330 ± 27	12.0 ± 0.6	3.03 ± 0.23	185
	StyleMamba	290 ± 23	10.5 ± 0.5	3.45 ± 0.25	152
	ArtBank	430 ± 36	15.8 ± 1.0	2.33 ± 0.17	280
	S2WAT	372 ± 31	13.0 ± 0.7	2.69 ± 0.20	210
	InstructG2I	385 ± 30	13.6 ± 0.8	2.60 ± 0.19	235
	PixWizard	412 ± 33	14.2 ± 0.9	2.43 ± 0.18	260
	MobileDiffusion	480 ± 34	8.8 ± 0.3	2.08 ± 0.15	128
	SnapFusion	950 ± 45	10.2 ± 0.5	1.05 ± 0.08	156
	DCLSDM	315 ±22	10.2 ±0.4	3.17 ±0.22	148
Summer2Winter	StyleGAN-NADA	480 ± 33	14.1 ± 0.8	2.08 ± 0.16	256
	ControlNet	346 ± 29	12.3 ± 0.6	2.89 ± 0.22	198
	StyleStudio	391 ± 31	12.9 ± 0.7	2.56 ± 0.19	224
	DEADiff	320 ± 26	11.6 ± 0.5	3.12 ± 0.24	185
	StyleMamba	282 ± 21	10.1 ± 0.4	3.55 ± 0.26	152
	ArtBank	420 ± 34	15.4 ± 0.9	2.38 ± 0.18	280
	S2WAT	364 ± 30	12.6 ± 0.6	2.75 ± 0.21	210
	InstructG2I	378 ± 29	13.2 ± 0.7	2.64 ± 0.20	235
	PixWizard	400 ± 32	13.8 ± 0.8	2.50 ± 0.19	260
	MobileDiffusion	460 ± 32	8.5 ± 0.3	2.17 ± 0.16	128
	SnapFusion	920 ± 42	9.8 ± 0.4	1.09 ± 0.09	156
	DCLSDM	308 ±21	9.9 ±0.3	3.25 ±0.23	148

Table 4 presents the comparison of LPIPS metrics for each model on five typical artistic style categories–Impressionism, Abstract Expressionism, Romanticism, Baroque, and Surrealism–in the WikiArt dataset. The data represents the average values and error ranges for each style subset. As a core metric for measuring perceptual similarity, the lower the LPIPS value, the better the visual consistency between the generated image and the target style, making it especially suitable for evaluating the subjective aesthetic feature transfer in artistic style transfer. From the data, it is evident that DCLSDM maintains the lowest LPIPS value across all selected artistic style categories, demonstrating its stable adaptability to diverse artistic styles. In Impressionism, DCLSDM achieves an LPIPS of 0.190, significantly lower than StyleGAN-NADA at 0.285, ArtBank at 0.268, InstructG2I at 0.232, and PixWizard at 0.225, indicating its ability to accurately capture the characteristic light and shadow play and color flow of Impressionism. For Abstract Expressionism, a complex style emphasizing emotional tension and non-representational expression, DCLSDM achieves an LPIPS of 0.220, which is 13.1% lower than the next best model ControlNet at 0.253 and also lower than InstructG2I at 0.267 and PixWizard at 0.258, validating its deep understanding and transformation capability for abstract visual language. In Romanticism, which emphasizes emotional rendering and poetic expression of natural landscapes, DCLSDM’s LPIPS of 0.198 is 19.5% lower than StyleStudio at 0.246, and also outperforms InstructG2I at 0.241 and PixWizard at 0.233, reflecting its precise transfer of subtle emotional styles. The Baroque style, characterized by extravagant decoration and strong contrasts, sees DCLSDM with an LPIPS value of 0.206, outperforming DEADiff at 0.247, S2WAT at 0.263, InstructG2I at 0.250, and PixWizard at 0.242, demonstrating its effective reproduction of dramatic visual effects typical of this style. In Surrealism, a style filled with fantastical elements, DCLSDM achieves an LPIPS of 0.213, 24.0% lower than StyleMamba at 0.280 and lower than InstructG2I at 0.256 and PixWizard at 0.248, showing its accurate capture of unconventional visual symbols. Overall, DCLSDM maintains a perceptual similarity advantage across different artistic movements and style transfer tasks with varying complexities. Its superiority is especially evident in categories with distinctive style features, such as Abstract Expressionism and Surrealism, confirming the effectiveness of its content-style decoupling mechanism in artistic style transfer scenarios. It can preserve the content structure while more accurately conveying the core visual features of the target style.

Table 4.

Comparison of LPIPS for different artistic style categories.

Artistic style category	StyleGAN-NADA	ControlNet	StyleStudio	DEADiff	StyleMamba	ArtBank	S2WAT	InstructG2I	PixWizard	DCLSDM
Impressionism	0.285 ± 0.026	0.218 ± 0.020	0.237 ± 0.022	0.229 ± 0.021	0.252 ± 0.023	0.268 ± 0.025	0.245 ± 0.022	0.232 ± 0.021	0.225 ± 0.020	0.190 ±0.017
Abstract expressionism	0.321 ± 0.030	0.253 ± 0.024	0.276 ± 0.026	0.264 ± 0.023	0.289 ± 0.027	0.298 ± 0.028	0.279 ± 0.025	0.267 ± 0.024	0.258 ± 0.023	0.220 ±0.020
Romanticism	0.293 ± 0.027	0.225 ± 0.021	0.246 ± 0.023	0.238 ± 0.022	0.260 ± 0.024	0.273 ± 0.026	0.255 ± 0.023	0.241 ± 0.022	0.233 ± 0.021	0.198 ±0.018
Baroque	0.305 ± 0.028	0.236 ± 0.022	0.259 ± 0.024	0.247 ± 0.022	0.272 ± 0.025	0.286 ± 0.027	0.263 ± 0.024	0.250 ± 0.023	0.242 ± 0.022	0.206 ±0.019
Surrealism	0.312 ± 0.029	0.242 ± 0.023	0.265 ± 0.025	0.253 ± 0.023	0.280 ± 0.026	0.291 ± 0.027	0.270 ± 0.025	0.256 ± 0.023	0.248 ± 0.022	0.213 ±0.020

Qualitative analysis

Figure 2 demonstrates the comparison of multi-style image generation results. From the transfer results of multiple content and style image pairs, DCLSDM shows superior style fusion and content preservation capabilities compared to methods like StyleGAN-NADA, ControlNet, InstructG2I, and PixWizard. Leveraging the content feature extraction module, the model accurately captures the geometric structure and semantic information of the content: in the train scene, the outlines of the tracks and the carriages are clearly distinguishable; in the flower case, the shape of the vase and the details of the flowers are fully preserved; during the style transfer of architectural and natural landscapes, the main structures such as the arch and trees remain undistorted by the style injection, which validates the content protection effect of the structural constraint module. In terms of style transfer, the style feature processing and dual-condition diffusion mechanism enable the model to efficiently decode style semantics. When faced with different art styles such as Impressionism and Realism, DCLSDM naturally integrates style textures into the content. For example, in the flower group, the generated image from Ours has a color distribution and texture density that more closely resemble the style image, both preserving the artistic characteristics of the “Pointillism” technique and maintaining the recognition of the flower subject. In the train and architectural scenes, the stylized images exhibit finer light-shadow transitions and color rendering, avoiding common issues in other models such as “style oversaturation leading to blurred content” or “color discontinuities.” Compared to InstructG2I, which often introduces graph-based artifacts, and PixWizard, which struggles with style-content balance in complex scenes, DCLSDM demonstrates the precise control of style expression through weighted Style Interpolation and independent condition injection. According to the quantitative metrics, DCLSDM achieves a lower LPIPS, indicating better style fidelity, and a higher SSIM, confirming more complete content structure preservation. These results are in line with the model design strategies of “content-style decoupling control” and “lightweight feature extraction optimization.” This shows that by decoupling content feature extraction and style feature processing and injecting dual-condition constraints in the latent diffusion process, the model is able to better preserve content structure while maintaining high style fidelity, surpassing InstructG2I’s graph-guided but often rigid style transfer and PixWizard’s multi-task but less focused style adaptation.

Fig. 2.

Qualitative comparison of the proposed model and baseline methods.

The results in Table 5 focus on different weighting schemes for the total loss function of the DCLSDM model, quantifying the performance differences between fixed weights and adaptive weights on two datasets using three core metrics. Regarding the performance of the fixed weights (), they exhibit optimal performance on both datasets. On the WikiArt dataset, the SSIM reaches , the LPIPS is as low as , and the FID is . On the Summer2Winter dataset, the SSIM further increases to , the LPIPS decreases to , and the FID is as low as . This result is fully consistent with the conclusion in the original paper that “this fixed weight is determined through grid search on the validation set and can balance structural preservation and style fidelity”, confirming its universal advantage in both artistic and scene style transfer tasks. The performance of the linear adaptive weights ( gradually increases from 0.3 to 0.8, and gradually increases from 0.1 to 0.3) shows obvious stage dependence. In the 1-50k iteration interval (, ), the SSIM values of the two datasets are only (WikiArt) and (Summer2Winter), the LPIPS values are as high as and , and the FID exceeds 27. This reflects the problems of insufficient content structure preservation and severe coupling between style and content under weak structural constraints and weak decoupling constraints. As the iteration progresses to the 100k-150k interval (, ), the performance improves to some extent (the SSIM of WikiArt increases to , and the LPIPS decreases to ), but it is still significantly lower than that of the fixed weights. Moreover, the FID of the two datasets is still higher than 24.9, indicating that although linear weight adjustment can alleviate some problems, the weight fluctuation leads to unstable model training, making it difficult to achieve the optimal balance between structure and style. The performance of the attention-based adaptive weights (dynamically allocating based on the content complexity extracted by Perceiver IO) is close to that of the fixed weights (on WikiArt, the SSIM is and the LPIPS is ; on Summer2Winter, the SSIM is and the LPIPS is ), but it does not exceed the fixed weights. Additionally, the content complexity judgment it relies on is completely based on the existing Perceiver IO module in the original paper, without adding any new calculation logic. This further demonstrates the advantage of fixed weights in “performance stability”–they can stably achieve the core goal of content-style decoupling on both types of datasets without relying on additional dynamic judgment mechanisms, which is highly consistent with the model design concept of “lightweight, efficient and controllable” in the original paper. In summary, the comparison results of different weighting schemes clearly show that the fixed weights () selected in the paper achieve the optimal balance in structural preservation, style fidelity, and generation quality. Their training stability and universality are superior to those of linear adaptive weights and attention-based adaptive weights. This provides a reliable parameter selection basis for the application of the model in scenarios such as new media content creation and cultural relic restoration, and also verifies the rationality of the loss function design in the original paper.

Table 5.

Quantitative performance comparison of DCLSDM with different weighting schemes.

Weighting scheme	Weight range	WikiArt			Summer2Winter
		SSIM	LPIPS	FID	SSIM	LPIPS	FID
Fixed weights (Proposed)		0.832 ± 0.023	0.195 ± 0.019	22.52 ± 1.78	0.842 ± 0.021	0.187 ± 0.018	21.34 ± 1.71
Linear adaptive weights	: 0.30.5; : 0.10.2	0.795 ± 0.028	0.231 ± 0.022	28.64 ± 1.91	0.802 ± 0.026	0.227 ± 0.021	27.81 ± 1.85
	: 0.50.65; : 0.20.25	0.808 ± 0.027	0.223 ± 0.021	27.15 ± 1.88	0.815 ± 0.025	0.220 ± 0.020	26.43 ± 1.82
	: 0.650.8; : 0.250.3	0.818 ± 0.025	0.212 ± 0.020	25.87 ± 1.82	0.825 ± 0.023	0.218 ± 0.019	24.93 ± 1.76
Attention-based adaptive weights	dynamic	0.830 ± 0.024	0.198 ± 0.020	23.15 ± 1.80	0.840 ± 0.022	0.192 ± 0.019	22.07 ± 1.74

The generated Fig. 3 vividly illustrates the model’s behavior under different combinations of loss function coefficients (weight for structural constraint loss ) and (weight for decoupling constraint loss ). Taking the Great Wall and architectural content images paired with corresponding artistic styles as examples, when and (weak constraints), the style exerts a strong dominance over the content generation. In the Great Wall image, the structural details like the outline become blurred, with style brushstrokes taking the lead, while in the architectural image, the style permeates extensively, causing distortion in the building’s form, reflecting extreme style expression at the cost of structural integrity. As increases to and to (balanced constraints), a natural integration of style and content emerges. The Great Wall retains clear architectural outlines and the building maintains intact morphology, with harmonious style rendering, well - suiting practical new - media creation scenarios that require a balance between style and content. When and (strong constraints), structural fidelity is prioritized. Fine details such as the bricks of the Great Wall and the building’s contours remain intact, and the style integration is more controlled, adhering closely to the structural framework while still imparting the artistic essence, which caters to scenarios like cultural relic restoration with high structural precision requirements. Overall, different coefficient combinations correspond to the balance changes between structural fidelity and style expression in style transfer. The balanced constraint performs optimally in general scenarios, demonstrating the model’s flexible adaptability in controlling style and content.

Fig. 3.

Stylization outcomes under varying loss coefficients.

Ablation studies

The ablation study results, integrated from Table 6 and Fig. 4, demonstrate that the full Dual-Condition Lightweight Style Diffusion Model (DCLSDM) achieves optimal performance across all evaluation metrics on both the WikiArt and Summer2Winter datasets. This validates the synergistic effectiveness of its core components. All performance differences between the full model and its variants are statistically significant with p < 0.05 or p < 0.01, confirming the genuine contributions of each component rather than random data fluctuations. When replacing the Perceiver IO content feature extractor with ViT, the visual results show that generated images lack fine structural preservation. Artwork outlines in WikiArt and mountain contours in Summer2Winter become blurred. Quantitatively, this translates to a 0.034 SSIM decrease, 0.036 LPIPS increase, 2.13 dB PSNR drop and 4.35 FID rise on WikiArt with p < 0.01. On Summer2Winter, SSIM falls by 0.037, LPIPS increases by 0.031 and FID climbs by 4.13 with p < 0.01. This consistency between visual degradation and quantitative declines confirms Perceiver IO’s superiority in balancing feature expression and computational efficiency–its cross-modal attention mechanism compresses content features into low-dimensional latent space without losing structural details, which aligns with the model’s lightweight design goal. Removing the weighted Style Interpolation module leads to disjointed style fusion in the visual results. Mixed artistic styles in WikiArt appear fragmented, and seasonal tone transitions in Summer2Winter lack coherence. Corresponding metrics show LPIPS increases by 0.025 and FID rises by 2.61 on WikiArt with p < 0.05. On Summer2Winter, LPIPS increases by 0.018 and FID rises by 2.55 with p < 0.05. This verifies the module’s role in enhancing style fidelity and diversity–its dynamic weighting strategy fuses multi-style features smoothly, avoiding the “style averaging” blur seen in other interpolation methods. The most severe degradation occurs in the group without ControlNet. The visual results reveal severe structural distortion. Artwork shapes in WikiArt are distorted, and tree outlines in Summer2Winter are unrecognizable. Quantitatively, this translates to a 0.068 SSIM decrease, 2.95 dB PSNR drop and 6.83 FID rise on WikiArt with p < 0.01. On Summer2Winter, SSIM falls by 0.071, PSNR drops by 2.55 dB and FID increases by 6.57 with p < 0.01. This proves ControlNet’s critical role in maintaining structural consistency–its trainable control module encodes edge and depth information into diffusion features, locking key geometric details during style transfer to avoid distortion. Using a shared attention path causes muddled content-style integration in the visual results. Style textures overwrite content details such as brushstrokes obscuring WikiArt flower shapes. Content also disrupts style as seen in the Summer2Winter mountain edges breaking winter tones. The metrics mirror this with a 0.05 SSIM decrease and 0.04 LPIPS increase on WikiArt with p < 0.01. On Summer2Winter, SSIM falls by 0.046 and FID rises by 4.49 with p < 0.01. This confirms the necessity of dual-condition independent paths–separate content and style interaction channels ensure each factor exerts control without mutual interference, realizing effective content-style decoupling. For the “Perceiver IO with Self-Attention” group, the visual results show reduced cross-modal feature capture. Text-described “snow-capped mountains” in Summer2Winter lose semantic alignment, and WikiArt image content misses fine details. The metrics record SSIM decreasing to 0.801 and 0.812, LPIPS rising to 0.227 and 0.211, FID climbing to 26.14 and 24.98 on WikiArt and Summer2Winter respectively with p < 0.05. In the “Average Pooling for ” group, the visual results show blurred edges. WikiArt contours soften, and Summer2Winter mountain ridges lose sharpness. Metrics show SSIM dropping to 0.805 and 0.818, LPIPS increasing to 0.216 and 0.203, FID rising to 25.73 and 24.35 on WikiArt and Summer2Winter respectively with p < 0.05. These confirm Perceiver IO’s cross-modal attention optimizes multi-modal capture by integrating text and image content information. Max pooling preserves edge structure by retaining sharp contour details. Both components support U-Net’s design to enhance content-style decoupling, structural consistency and style fidelity. Collectively, the ablation results validate that each core component of DCLSDM addresses a key research goal. Perceiver IO ensures lightweight content feature extraction to support efficiency. Weighted Style Interpolation enhances style diversity to adapt to multi-style demands. ControlNet preserves structural consistency to solve distortion issues. Independent attention paths achieve content-style decoupling as the core goal. The synergistic effect of these components confirms DCLSDM’s ability to balance generation quality, controllability and efficiency–fulfilling the technical demands of multi-style image generation in new media scenarios.

Table 6.

Ablation Study Results of DCLSDM and Its Variants on WikiArt and Summer2Winter Datasets. Note: Statistical significance is determined via two-tailed independent samples t-tests compared to the Full Model (*p<0.05, **p<0.01).

Model name	WikiArt					Summer2Winter
	SSIM	LPIPS	PSNR (dB)	FID	p-value	SSIM	LPIPS	PSNR (dB)	FID	p-value
Full model	0.832 ± 0.023	0.195 ± 0.019	32.28 ± 1.05	22.52 ± 1.78	–	0.842 ± 0.021	0.187 ± 0.018	32.63 ± 1.01	21.34 ± 1.71	–
ViT replaces Perceiver IO	0.798 ± 0.025	0.231 ± 0.022	30.15 ± 1.12	26.87 ± 1.93	<0.01**	0.805 ± 0.024	0.218 ± 0.020	31.22 ± 1.08	25.47 ± 1.86	<0.01**
Without weighted style interpolation	0.815 ± 0.024	0.220 ± 0.021	31.02 ± 1.09	25.13 ± 1.89	<0.05*	0.827 ± 0.023	0.205 ± 0.019	31.85 ± 1.06	23.89 ± 1.82	<0.05*
Without ControlNet	0.764 ± 0.028	0.248 ± 0.023	29.33 ± 1.15	29.35 ± 2.01	<0.01**	0.771 ± 0.026	0.232 ± 0.021	30.08 ± 1.13	27.91 ± 1.94	<0.01**
Shared attention path	0.782 ± 0.026	0.235 ± 0.022	30.57 ± 1.11	27.62 ± 1.96	<0.01**	0.796 ± 0.025	0.212 ± 0.020	31.15 ± 1.09	25.83 ± 1.88	<0.01**
Perceiver IO with self-attention	0.801 ± 0.025	0.227 ± 0.022	30.85 ± 1.10	26.14 ± 1.92	<0.05*	0.812 ± 0.024	0.211 ± 0.019	31.56 ± 1.07	24.98 ± 1.85	<0.05*
Average pooling for	0.805 ± 0.024	0.216 ± 0.020	30.92 ± 1.08	25.73 ± 1.90	<0.05*	0.818 ± 0.023	0.203 ± 0.018	31.72 ± 1.05	24.35 ± 1.83	<0.05*

Fig. 4.

Visual comparison of ablation experiments for DCLSDM variants on datasets.

Table 7 provides direct quantitative validation of the model’s content-style decoupling effect from three core dimensions: feature correlation, content preservation, and style fidelity. In terms of content-style feature mutual information, the full model achieves mutual information values of 1.23 ± 0.11 on the WikiArt dataset and 1.18 ± 0.10 on the Summer2Winter dataset. In contrast, the model without decoupling constraint loss shows significantly higher mutual information values, reaching 1.99 ± 0.15 on WikiArt and 1.83 ± 0.13 on Summer2Winter. The lower mutual information of the full model demonstrates that the decoupling constraint loss effectively reduces the correlation between content and style features, laying a foundation for their independent control at the feature level. For the structural preservation verification under fixed content and multi-style transfer, 3 typical content images were used, with 5 distinct styles applied to each content. The full model achieved an average SSIM of 0.84 ± 0.02 on WikiArt and 0.85 ± 0.02 on Summer2Winter between the generated images and the original content. Even the minimum SSIM values remained above 0.82 ± 0.03 on and 0.83 ± 0.02 on Summer2Winter . This indicates that the model can stably retain the core structure of the content despite significant style differences, verifying the effectiveness of the content control path in ensuring structural integrity. In terms of style fidelity for fixed content and multi-style transfer, the full model achieved an average LPIPS of 0.201 ± 0.018 on WikiArt and 0.192 ± 0.017 on Summer2Winter between the generated images and the target styles. The low LPIPS values confirm that the model does not sacrifice the accuracy of style expression while stably preserving content structure, thereby achieving effective content-style decoupling and meeting the core requirement of “constant content with flexible style switching.”

Table 7.

Direct quantitative validation of content-style decoupling effect.

Validation index	Model setting	WikiArt dataset	Summer2Winter dataset
Content-style feature mutual information	Full model (with Decoupling Loss)	1.23 ± 0.11	1.18 ± 0.10
	Without decoupling loss	1.99 ± 0.15	1.83 ± 0.13
Fixed content + Multi-style SSIM	Average SSIM (5 Styles)	0.84 ± 0.02	0.85 ± 0.02
	Minimum SSIM (5 Styles)	0.82 ± 0.03	0.83 ± 0.02
Fixed content + Multi-style LPIPS	Average LPIPS (5 Styles)	0.201 ± 0.018	0.192 ± 0.017

Fixed Content + Multi-Style” test uses 3 typical content images (flower vase, mountain landscape, architectural outline) and 5 styles (Impressionism, Baroque, Surrealism, Minimalism, Romanticism).

Discussion

Key findings and limitations

The proposed Dual-Conditional Lightweight Style Diffusion Model demonstrates notable strengths. Its dual-conditional control architecture integrates content feature extraction via Perceiver IO and independent style injection to achieve effective content-style decoupling. This ensures precise preservation of core content structures during style transfer, outperforming models like StyleGAN-NADA, ControlNet, InstructG2I, and PixWizard in maintaining content integrity while infusing diverse styles. Leveraging latent diffusion and lightweight components reduces computational complexity, leading to faster inference and lower memory usage than other models. For example, InstructG2I uses approximately 18% more memory and PixWizard has about 25% longer inference time. This makes DCLSDM especially suitable for resource-constrained scenarios, such as mobile applications. The weighted Style Interpolation and multi-modal input support enable flexible adaptation to various styles, including abstract art, with higher fidelity.

Synthesizing experimental findings from the WikiArt and Summer2Winter Yosemite datasets, these advantages are quantitatively validated. DCLSDM achieves an SSIM of 0.832–0.842 compared to 0.796–0.822 for InstructG2I and PixWizard, and a LPIPS of 0.187–0.195 compared to 0.201–0.232 for InstructG2I and PixWizard, while maintaining fewer parameters (148M vs. 235M and 260M for the two baselines). This confirms that its decoupling design and lightweight framework effectively balance generation quality and efficiency. Ablation studies further reinforce this–removing Perceiver IO reduces SSIM by 0.034, while omitting dynamic style interpolation increases LPIPS by 0.025, proving the synergistic value of each core module. These findings validate that independent conditional paths, rather than shared attention, provide a viable solution to the long-standing content-style coupling issue in latent diffusion models. This offers a new technical paradigm for multi-style generation tasks. Additionally, DCLSDM’s balance of performance and efficiency bridges the gap between academic research and industrial application. It demonstrates that diffusion-based style generation models can be adapted for resource-constrained scenarios, such as mobile new media, which was previously challenging for heavyweight models like ArtBank, as well as InstructG2I and PixWizard (both with higher computational overhead). However, the model does have limitations. While DCLSDM excels in handling a wide range of styles, it can struggle when the input style is significantly different from the ones present in the training data–similar to InstructG2I’s challenges with unseen graph structures and PixWizard’s limitations in rare style categories. In these cases, the model might not fully capture the target style, resulting in suboptimal style transfer. Additionally, the model’s performance is highly sensitive to input quality. For example, ambiguous or poorly defined input descriptions, or low-resolution content images, can lead to degraded results where the content structure is not well preserved, or the style transfer is inconsistent–an issue that also affects the instruction-driven capabilities of PixWizard. These challenges are typical of models that rely on well-defined, high-quality inputs, and while they do not constitute widespread failure, they point to areas for improvement, particularly in expanding the model’s robustness to diverse input types.

The model also lacks an efficient real-time interactive interface, limiting its usability in dynamic design workflows. This is an area where future development could make the model more practical for creative industries that require interactive and immediate style transfer.

Future improvements

Future work will focus on addressing the limitations related to rare or under-represented styles and input quality sensitivity. Specifically, to improve generalization to rare styles, we will implement meta-learning and few-shot learning techniques to allow the model to adapt to new, unseen styles with limited samples. This will enable better handling of styles that are not well-represented in the training data. Additionally, to improve the model’s robustness to input quality variability, we will explore advanced preprocessing techniques such as adaptive denoising and conditional image enhancement models. These methods will dynamically adjust based on input quality, ensuring that the model can handle ambiguous descriptions or low-resolution images more effectively, reducing the performance degradation in such cases. Moreover, we plan to incorporate real-time interactive adjustment capabilities, allowing users to modify style intensity or content-style balance dynamically while receiving instant feedback. User studies and preference scores will be incorporated to validate the model’s performance across diverse user preferences, ensuring that DCLSDM remains adaptable to different real-world applications. These improvements aim to enhance DCLSDM’s versatility, making it suitable for a wider range of real-world scenarios, including dynamic creative workflows and resource-constrained environments like mobile devices.

Conclusion

This study proposes DCLSDM, a multi-style image generation model based on latent diffusion architecture, to address core challenges including insufficient content-style decoupling, high computational costs, and structural distortion in existing methods. By integrating dual-conditional control, lightweight feature extraction, and structural constraint mechanisms, DCLSDM achieves independent control of content and style while maintaining efficiency. Experimental results on public datasets demonstrate that DCLSDM outperforms representative academic and industrial models in generation quality and computational efficiency, making it suitable for resource-constrained scenarios. DCLSDM provides a practical solution for multi-style image generation in content creation and digital art production. Future work will focus on enhancing generalization to rare styles, improving robustness to low-quality inputs, and developing real-time interactive capabilities.

Author contributions

Kaiyan Chu contributed to the conceptualization, methodology, and formal analysis of the study, as well as writing and revising the manuscript. Yu Shang was involved in data collection, investigation, and software development. Lingrui Zhang assisted in the analysis of the results and provided critical revisions to the manuscript. Haiyu Yuan supervised the project, provided guidance on methodology, and reviewed the manuscript. All authors have read and agreed to the published version of the manuscript.

Data availability

The datasets used in this study are publicly available. The Summer2Winter Yosemite dataset can be accessed at https://www.kaggle.com/datasets/balraj98/summer2winter-yosemite, and the WikiArt dataset is available at https://hf-mirror.com/datasets/williamberman/wikiart.

Declarations

Competing interests

The authors declare no competing interests.