ShadowFormer++: multi-scale shadow priors and diffusion-guided refinement for high-fidelity shadow removal

Abstract

Shadows in images cause significant challenges for computer vision applications as they degrade image quality and hinder tasks such as object detection, visual tracking, and semantic segmentation. Existing shadow removal methods often struggle as they fail to localize shadow regions accurately without suppressing non-shadow areas; they tend to lose fine-grained textures and illumination consistency across boundaries, and they also demand heavy computation that limits their use in real-time or resource constrained systems. To address these limitations, we propose a novel framework called ShadowFormer++, which uses a combination of a transformer and diffusion model for efficient shadow removal. Our architecture integrates Multi-Scale Local Shadow Perception Module (MS-LSPM) for robust local shadow feature extraction, Shadow-Aware Transformer Encoder (SATE) for global context modelling and structure preservation, and a Diffusion-Inspired Refinement Module (DIRM) for progressive, fine-grained shadow-free image reconstruction. Extensive experiments on ISTD, ISTD+, and SRD datasets show that ShadowFormer++ achieves a Mean Absolute Error (MAE) of 3.79, a Peak Signal-to-Noise Ratio (PSNR) of 33.58 dB, and a Structural Similarity Index (SSIM) of 0.972, which is better than state-of-the-art methods such as ShadowFormer, SpA-Former, and Diff-Shadow. Our approach balances computational efficiency with superior shadow removal quality, making it suitable for real-time applications in robotics, autonomous systems, and augmented reality.

Introduction

Shadows are an inevitable byproduct of light object interactions, although they play an important role in visual perception, they present significant challenges in computer vision tasks such as object detection, scene understanding, and remote sensing analysis¹. Shadows often occur when a surface lacks direct illumination, which has attracted considerable interest in studies². The shadows, shielded from direct light, help in obtaining lighting direction, scene geometry, and color parameters. However, they can blur objects, affecting computer vision tasks such as object detection and semantic segmentation³. Hard and soft shadows are two main types that differ in sharpness and transition from light to dark. Hard shadows have sharp, clearly defined edges, created by a small, direct light source like a flashlight or the sun. Soft shadows have blurred, diffused edges and a gradual transition, caused by a large, diffused light source or when ambient light fills in the shadows¹. Shadow removal is a basic image enhancement technique that eliminates shadow distortions caused in images, restoring detail and color consistency⁴. Shadow removal has long been a challenging subject that affects many applications such as scene interpretation, augmented reality, and image enhancement. Utilizing fundamental physics principles to remove the shadow it is difficult to generalize and remain durable in real-world scenarios⁵. Shadow removal in images is the process that removes shadows while keeping as much of the underlying context as possible. Consequently, achieving high-quality shadow-free results remains a challenge. There are still issues persistent in the field of shadow removal including residual borders in the recovered images, artifacts in the recovered shadow-free images, or a lack of smooth color compatibility between the shadow and non-shadow areas. As a result, research is being done to remove these shadow artifacts so that the image appears more realistic⁶. The shadow removal process must handle the detection of shadows, cleaning up their edges, and fixing lighting all at once in all types of scene⁷. The shadows in the images are highly dependent on lighting, what the surface is made of, and the shape of what blocks the light as it can cause issues such as maintaining the color consistency where the shadow was and the rest of the image, adding artifacts around the edges of the shadow⁸, and losing minute details in the image⁹.

Deep learning techniques have significantly advanced computer vision, improving applications such as video monitoring systems and self-driving cars¹⁰. Recent advances in shadow removal have benefited significantly from deep learning techniques, as neural networks have demonstrated an improved ability to distinguish between shadowed and non-shadowed regions¹¹. Shadow removal, a particularly intricate task due to the need to reconstruct irregularly shaped shadow regions, has seen deep learning emerge as the leading approach recent studies shows this change¹² In particular, strategic incorporation of shadow masks allows precise differentiation between shadowed and non shadowed areas, which has been instrumental in the success of deep learning models for shadow removal¹³. Although deep learning excels at shadow removal, it often does not mimic real-world shadow behavior. Many current deep learning techniques have improved by incorporating elements or related to physics.

Traditional shadow removal methods relied mainly on intrinsic image decomposition and illumination modeling, with the aim of decoupling surface reflectance from shading components¹⁴. These methods performed well in straightforward scenarios when dealing with hard shadows, but struggled under complex lighting conditions or when dealing with soft shadows. With the advent of deep learning, Convolutional Neural Network (CNN) methods, such as DeshadowNet¹⁵ and ST-CGAN¹⁶, have demonstrated significant advancements by implementing end-to-end frameworks for shadow removal. However, these models primarily focus on localized features, which limits their ability to capture the global illumination context essential for accurate restoration. Later studies investigated the attention mechanism and used adversarial training to get the shadow edges right. For example, Mask-ShadowGAN¹³ used adversarial learning with shadow masks, and DSC¹⁷ used a spatial context that knows the direction to model shadow areas. Newer transformer models, such as ShadowFormer⁷ , used self-attention to model the long distance link and showed better generalization across datasets. Diffusion methods such as Diff-Shadow¹⁸ and Shadow Diffusion¹⁹ also used production priors to refine structure consistency, but at the cost of high computing due to iterative denoising¹⁸. Figure 1 illustrates these categories of the deep learning based shadow removal algorithm along with some models under each category. Despite recent advances, shadow removal techniques remain constrained by several factors such as accurate shadow localization, frequent loss of grained textures and exhibit illumination inconsistencies, and substantial computational overhead. Current models often have issues with shadow patterns, where hard and soft shadows are in the same scene. Existing models either focus on removing shadow but miss small addition of the artifacts, or while focusing on the artifacts they miss the illumination details. Achieving both high accuracy and real-time performance remains difficult, particularly for applications such as robotics and surveillance that require immediate processing. Therefore, there is a strong need for a shadow removal solution that maintains a good balance between accuracy, visual realism, and computational efficiency. As current methods require a compromise between fine detail and overall image consistency.

Fig. 1.

Deep learning shadow removal techniques.

The novelty of this work is the introduction of ShadowFormer++, a new approach to shadow removal that strategically integrates several advanced components. Our model features a learnable Multi-Scale Local Shadow Perception Module, which extracts robust shadow priors across multiple receptive fields. These priors guide a Shadow-Aware Transformer Encoder, enabling the model’s self-attention mechanism to preserve both global structure and fine details. Additionally, ShadowFormer++ includes a lightweight Diffusion-Inspired Refinement Module that emulates iterative denoising, enabling high-fidelity and realistic shadow-free image reconstruction. The major contributions of this work are as follows.

• We proposed ShadowFormer++, a transformer-based model designed for effective shadow removal from images.

• The framework incorporates denoising principles inspired by diffusion processes to recover fine-grained structural details.

• ShadowFormer++ consists of three key modules: a Multi-Scale Local Shadow Perception Module (MS-LSPM), a Shadow-Aware Transformer Encoder (SATE), and a Diffusion-Inspired Refinement Module (DIRM), enabling efficient and accurate shadow elimination.

• We evaluated ShadowFormer++ on three widely used datasets that are ISTD, ISTD+, and SRD where it outperformed leading methods such as ShadowFormer⁷, SpA-Former²⁰, and Diff-Shadow¹⁸, achieving superior Mean Absolute Error (MAE), Structural Similarity Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR) scores.

• An ablation study is performed to evaluate the contribution of each component of the proposed ShadowFormer++ framework and to demonstrate the effectiveness of the individual modules.

The subsequent sections of this paper are organized as follows: “Literature review” provided an overview of the related literature, “Proposed model” described the proposed model, “Experimental setup” discussed the implementation details and datasets used, “Result and discussion” presented the results and discussion, and finally “Conclusion” concluded the paper.

Literature review

In this section, some of the recent work on shadow removal techniques is briefly discussed. Shadow removal has progressed from physical model based and illumination prior methodologies, which are constrained in intricate scenes, to deep learning techniques that utilize paired or unpaired datasets for enhanced generalization and precision. Shadow removal, which aims to eliminate as much visual semantic information that the shadow in the image obscures or hides, is one of the most crucial image restoration tasks. In addition to the varied intricacy of shadow depth and color caused by arbitrary background elements, the shadow removal task often involves variations in lighting^21,22. Removing shadows is a complex issue because one has to deduce the appearance of the obscured regions without really knowing the underlying structure. This is the reason why researchers have come up with some smart ways to solve this by integrating how light works with state-of-the-art computational intelligence²³. Conventional shadow removal mostly used math models of how shadows form. Barrow et al.²⁴ introduced the concept of intrinsic images to decompose a scene into its constituent reflectance and illumination components. This basic idea showed that you can get images without shadows by finding changes in scene lighting but keeping the reflective surfaces the same. Finlayson et al.⁵ developed shadow-invariant images by projecting image colors onto directions orthogonal to illumination changes. Their approach utilized the invariant image to detect shadow regions and removed shadows by zeroing gradients at shadow edges. However, this method was limited to scenes with simple illumination conditions⁵.

Deep learning based methods have shown great potential but can have issues with overfitting if there is not enough good data to train on. It really needs a lot of quality data to work well^25,26. Deep learning methodologies have become ubiquitous in computer vision, consistently outperforming conventional model-driven techniques²⁷. When it comes to shadow removal using deep learning methods, there are four main categories. The first uses CNNs, such as DeshadowNet¹⁵, BMNet²⁸, and DSC¹⁷. These methods extract local details using layers; this helps locate and eliminate shadows.The second uses Generative Adversarial Network (GAN) , such as ST-CGAN¹⁶, DHAN²⁹, MaskShadowGAN¹³, and DC-ShadowNet³⁰. These models learn from data to create shadow-free images that look real.The third set of methods, such as ShadowFormer⁷, ShadowMaskFormer³¹, and SpA-Former²⁰, uses Transformers. Transformers are good at gathering global background information through self-attention, which makes them very good at modeling long-range dependencies³². These methods employ self-attention to get a sense of the overall scene, which helps them handle shadow edges and slight changes in lighting better. The diffusion-based models, such as DiffShadow²⁸ and Shadow Diffusion¹⁹, which gradually improve the image by removing noise. Some hybrid approaches such as FieldNet¹ and ShadowGANFormer³³ are used as transformers along with CNN and GAN, respectively. Shadow removal research has moved forward, going from hand-designed methods to deep learning, and now to models using diffusion and transformers. The above mentioned models are discussed in the following subsection.

CNN-based approach

Early deep learning work set the stage by viewing shadow removal as a direct pixel translation task. Qu et al.¹⁵ came up with DeshadowNet, one of the first complete deep learning methods to remove shadows from end to end. DeshadowNet¹⁵ tackled the main issue of bringing shadow detection and removal together in one set-up using a multi-context embedding technique. The design had three networks that worked together: a Global Localization Network (G-Net) that pulled out shadow features describing the overall structure and top-level meaning, an Appearance Modeling Network (A-Net) that extract appearance features from the earlier layers, and a Semantic Modeling Network (S-Net) that got semantic info from the deeper layers.The great thing about DeshadowNet¹⁵ was that it could directly learn how to go from shadow images to shadow matte without needing separate shadow detection actions. This eliminated any need to clean up the image later and made the system flexible enough to handle shadows with different penumbra widths in different semantic spots. BMNet²⁸ used two-way mapping with clear mask guidance to make shadow and non-shadow areas match up. However, mask knowledge was needed, making it difficult to work in open settings. It matched shadow removal and shadow creation in a single setup with shared settings. Shadow removal and creation are related concepts that can inform each other. BMNet used both forward and backward mapping with inputs such as shadow masks and color maps²⁸. This two-way approach helped the network learn stronger representations by taking advantage of how removal and creation complement each other. The system outperformed previous methods with fewer configurations, suggesting that joint learning setups can be valuable²⁸.

GAN-based approach

Wang et al.¹⁶ proposed ST-CGAN using a stacked conditional GAN setup to learn shadow detection and removal at the same time, which helped it to work better with different types of shadow. ST-CGAN¹⁶ was the first system to learn both shadow detection and removal from start to finish. It had two conditional GANs stacked on top of each other. The first part created shadow detection masks from shadow images, while the second part used both the shadow image and the mask predicted to create images without shadows. This stacked architecture represents a novel approach to multi-task learning, departing from conventional multi-branch methods. The system used adversarial training to understand the scene as a whole for both shadow detection and removal. ST-CGAN¹⁶ demonstrated robust performance in both shadow detection and removal across multiple datasets, validating that joint learning frameworks significantly enhance the model’s contextual understanding of shadows. Later studies first focus on hierarchical feature learning. The Dual Hierarchical Aggregation Network (DHAN)²⁹ to fix ghost shadow removal issues. DHAN²⁹ had several growth-dilated convolutions without downsampling. It grouped multi-context features for attention and prediction in steps. To improve performance, DC-ShadowNet³⁰ added dual consistency learning, which ensured that spatial and color consistency was maintained between the predicted shadow-free areas and the actual ground truth. It could manage hard and soft shadows. The method added a shadow/shadow-free domain classifier to the generator and discriminator parts. This lets them focus on shadow areas. DC-ShadowNet³⁰ added new unsupervised losses based on shadow-free chromaticity, shadow-tough perceptual features, and boundary smoothness. It was the first unsupervised way to firmly remove shadows for both hard and soft shadows in single photos. The system also allows you to train during testing to get even better results³⁰. Alongside CNN improvements, mask-guided adversarial systems became popular. Mask-ShadowGAN¹³ made training possible without paired data by creating and improving shadow masks during adversarial learning. This new approach allowed shadow removal without paired data, but relied on good mask quality and could produce errors in areas with a lot of texture. The main new idea was learning to make shadow masks and remove shadows at the same time to get the best overall result. The method set up cycle-consistency rules between shadow-free and shadow-free areas using mask-guided creation, which solved the main problem that mapping between shadow-free and shadow-free images is not straightforward. Mask-ShadowGAN¹³ did about as well as supervised methods, but it only needed unpaired shadow and non-shadow images. Beyond dedicated shadow removal, recent advancements in attention-guided generative models for related image restoration tasks provide significant technical context. For instance, frameworks such as SARain-GAN³⁴ and LumiNet³⁴ demonstrate the efficacy of integrating spatial and multi-scale attention mechanisms within GAN architectures to handle localized degradations like rain streaks and backlit underexposure. These works highlight a growing trend toward using targeted attention to restore illumination and structural fidelity, a strategy that aligns with our use of shadow-aware perception modules.

Transformer-based approach

Recently, transformer models have shown they are really good at understanding long-distance links in data. ShadowFormer⁷ leverages global self-attention to maintain spatial consistency across the image, thereby significantly enhancing visual fidelity. This method uses a multiscale channel attention setup to capture overall data step by step. It also has a Shadow-Interaction Module (SIM) with Shadow-Interaction Attention (SIA) at a key point to figure out how shadow and non-shadow areas relate to each other⁷. Zhang et al.²⁰ came up with SpA-Former, a transformer that does shadow detection and removal all at once. Instead of doing it in two steps such as before, SpA-Former²⁰ learns how to directly turn images with shadows into shadow-free images. It does not need to find the shadows first. The way it is built combines transformer layers with Fourier transform blocks and a special spatial attention setup²⁰. Wan et al.³⁵ propose a model CRFormer to improve the performance this by enabling unidirectional interaction, directing information from non-shadow regions, which are abundant in color and illumination priors, to shadow regions in need of restoration, as opposed to generic global self-attention. This focused attention not only improves the restoration of structural details, but also ensures that the recovered areas fit the rest of the image³⁵. ShadowMaskFormer³¹, which is based on a transformer. It uses a special mask-augmented patch to help remove shadows. Instead of changing how attention works inside transformer blocks, they put shadow data in the system early on using special patch embedding³¹. The mask-augmented patch embedding mixes in shadow data and helps the model learn more about shadow areas³¹. This leads to better shadow removal without using too much computing power.

Diffusion-based approach

In addition to CNNs and transformers, researchers have investigated diffusion models to remove shadows. For example, ShadowDiffusion¹⁹ uses a step-by-step process to clean up images, producing good looking results, but it takes a long time. Also, Diff-Shadow¹⁸ builds on this idea by adding conditions to make the images even more true to life but still needs too much computing power to work in real time. Diff-Shadow¹⁸ is a diffusion model that uses global and local information to remove shadows from images. It brings together the detailed synthesis of diffusion models with global context to restore images without shadows²⁸.The model has two parallel UNets: one focuses on local areas to estimate noise, and the other creates a low-resolution version of the image without shadows²⁸.

Hybrid approach

Saleh et al.¹ proposed a model, FieldNet , which removes shadows in field robotics, even with low computing power. It works well with different types of light and shadows. But because it learns from synthetic training data, it might not work as well when the real world scenario or when it encounters unusual textures. ShadowGAN-Former³³ uses some GAN-driven learning and transformers – to eliminate shadows and make the image more consistent. It has Shadow Attention Reweight module and efficient attention mechanisms that removes shadows biased on the global context.

Despite these developments, every approach has drawbacks. Diffusion models can be time-consuming, transformers can miss subtleties, CNNs have trouble with context, and mask-guided methods need additional data. We introduce ShadowFormer++, a transformer blend that combines multiscale shadow information with a simple attention process and a quick refinement akin to diffusion models in order to address these problems. In shadow removal studies, this approach strikes a balance between speed and accuracy by preserving structure and texture while working efficiently.

Proposed model

In this section, we have presented our proposed shadow removal framework, ShadowFormer++. ShadowFormer++ aims to solve three main problems in shadow removal research that are shadow variability, model misinterpretation and Computational Inefficiency. Shadows can vary greatly in size and edge softness, ranging from sharp to diffuse boundaries. Traditional CNN-based models struggle to manage this variability, often resulting in incomplete or imperfect shadow removal. Although transformer-based methods excel at modeling dependencies within an image, they frequently lack explicit shadow-specific knowledge. This limitation can cause them to misinterpret lighting variations as changes in object shape, leading to less effective shadow elimination. While diffusion and generative adversarial techniques are capable of producing high-quality, realistic images, their substantial computational demands make them unsuitable for real-time applications. By addressing these challenges, ShadowFormer++ provides a more robust, accurate, and efficient solution for shadow removal. ShadowFormer++ employs a unified multi-branch architecture that combines lightweight refinement, global reasoning, and shadow priors into a single framework in order to solve these problems. Unlike previous methods that treat shadow removal as a purely image-to-image translation problem, our approach explicitly encodes shadow-specific cues at multiple scales and guides the network to reason about both structural and contextual illumination relationships. By combining transformer-based global understanding with a multi-size shadow module and a Diffusion Inspired Refinement Module (DIRM), ShadowFormer++ balances accuracy and realism while being efficient.

ShadowFormer++ aims to maintain a balance between maintaining the consistency of the lighting overall and preserving local details. This was not as well done by earlier shadow removal systems, which limited their ability to adjust to various circumstances. ShadowFormer++ employs a mix-and-match design in contrast to transformer-based models, which obtain the global view but lose fine details and consume more processing power, or standard CNN setups, which perform well at identifying local details but have trouble seeing the larger picture. The MS-LSPM, the SATE, and the DIRM make up its three components. A straightforward decoder completes the assembly. These parts work together to not only remove shadows but also fix the lighting so that the final image looks realistic. In the following subsections, we discuss each module in detail.

Preprocessing

In our proposed shadow removal model, preprocessing is the first stage. In preprocessing, we resize every RGB image and its shadow mask to a fixed resolution for each color channel, as Vision Transformers expect fixed-size inputs and standardizing to 224224 allows direct transfer learning. We use bilinear interpolation for RGB images and nearest-neighbor for shadow mask to ensure efficient batching and GPU utilization for subsequent stages. The resizing of images makes our model robust and compatible to different sizes of images. The size of the input images are pixels as the input has RGB images. After resizing our images we normalize the images using Min-Max Scaling ,so that the pixel value lies between 0 and 1. For the shadow image (which has 3 color channels and pixel values ranging from 0 to 224), the goal is to convert the integer pixel values into floating-point numbers between 0.0 and 1.0. For the shadow mask (which is typically a binary image where shadow regions are 255 and non-shadow regions are 0), the goal is to ensure the mask contains only 0.0 and 1.0 floating-point values. Normalization stabilizes gradient magnitudes and accelerates convergence. To perform normalization, we have used Eq. (1).

1

Where P is the pixel value.

Given the input image and its corresponding shadow mask , the preprocessing module produces the preprocessed shadow image and the preprocessed shadow mask .

Multi-scale learnable shadow prior module (MS-LSPM)

The MS-LSPM is used to specifically remove shadow information. Instead of just using a regular encoder, this module uses 33, 55, and 77 convolution layers allows the the model to simultaneously capture fine local details, medium size context, and global information to handle shadows of varying sizes and illumination. It also uses a attention guided mask generator that produces a shadow probability map which is called shadow prior, guiding the network to focus on shadow regions while preserving fine details and boundaries. Given an input image and shadow mask , the feature extraction is performed using Eq. (2).

2

Where represents a convolutional transformation with kernel size . Each branch produces features that are sensitive to edges, textures, and large shadow blobs, respectively. Here i represents the number of iterations which is 1,2,3 as we are using three different kernel sizes.

These features are concatenated as shown in Eq. (3)

3

Where are feature extracted by the convolutional transformation with kernel size respectively, and projected into a unified embedding space as shown in Eq. (4).

4

Where is a projection matrix and is the projection bias.

To explicitly localize shadows, MS-LSPM introduces an attention-guided mask generation step that uses a sigmoid function that ensures that the values are in the range . This is achieved using Eq. (5).

5

Where is the sigmoid function, is the attention weight and is the attention bias. The sigmoid activation produces a soft probability map in the range [0,1], and shadow localization is learned end-to-end via the Binary Cross Entropy (BCE) loss, without image-specific threshold tuning.

Finally, the shadow prior is formed by element-wise multiplication of the embedding and its corresponding attention mask, yielding feature maps that emphasize shadow regions while suppressing non-shadow content as shown in Eq. (6).

6

MS-LSPM creates flexible, adaptable priors that learn and adapts to changes with different scenes, such as indoor lighting or outdoor obstructions.

Shadow-aware transformer encoder (SATE)

The SATE in ShadowFormer++ is designed to help the network’s attention mechanism by using shadow priors. This enables the network process shadow areas more precisely while still accurately reconstructing global illumination. Regular vision transformers handle the entire image in the same same manner, but SATE uses shadow information from the MS-LSPM module to pay more attention to areas where shadows have a big impact. The SATE takes two inputs the preprocessed image and the shadow priors .The preprocessed image is fed into the patch embedding module, where it is divided into non-overlapping patches to patches yielding tokens for a image, each patch is then flattened and linearly projected into a -dimensional feature space, resulting in patch embeddings as shown in Eq. (7), which are subsequently passed to the SATE module for contextual feature learning.

7

Where represents the initial patch embeddings and is chosen following standard Vision Transformer configurations to balance model capacity with computational efficiency. Here, Flatten(.) denotes a tensor reshaping operation that converts spatial feature maps into token sequences for self-attention computation.

The SATE employs 12 layers for efficient model capture, balancing expressiveness and computational efficiency, and ensuring practical inference times for real-world shadow removal applications. This depth aligns with successful vision transformer architectures known for effectively modeling complex image relationships without excessive overfitting or training instability. Each layer integrates explicit shadow guidance into multi-head self-attention. Within the -th layer, the following operations are performed on the input tokens .

Linear projections

Tokens are projected into Query, Key, and Value spaces using learnable matrices , initialized via Xavier uniform initialization. Multi-head attention employs parallel heads, each with head dimension , allowing distinct attention patterns for different shadow characteristics.

Shadow-biased self-attention

The shadow prior map , produced by MS-LSPM, is downsampled and reshaped to a bias matrix . This alignment is necessary to match the token embedding space used in the transformer, as the transformer processes the image as a sequence of tokens, each corresponding to a patch.

Self-attention is computed as shown in Eq. (8).

8

where is a learnable scalar initialized to 0.1. The term prevents score explosion, while biases focus to shadow-affected tokens without discarding global context.

Layer normalization and MLP

The attention output undergoes layer normalization with , followed by a position-wise feed-forward network as shown in Eq. (9).

9

where , , and are learnable biases. The GELU (Gaussian Error Linear Unit) activation function is preferred in models because of its smooth, non-monotonic curve, which probabilistically weights inputs instead of harshly truncating negative values like ReLU. The hidden dimension provides ample capacity for non-linear feature mixing, essential for recovering fine-grained shadow boundaries.

Residual connection

A skip connection adds the layer input to the MLP output ensuring stable gradient flow and preserving low-level features throughout the 12-layer stack. The layer-wise update rule is shown in Eq. (10).

10

Where MLP() is the Multi Layer Perceptron, LN() is the layer normalization, SA is the self-attention and is the input token sequence at layer l.

SATE helps ShadowFormer++ keep structural accuracy, get back dropped surface details, and make reliable shadow-free rebuilds across different sets of information. The internal processing and layer architecture of SATE is illustrated in Fig. 3.

Fig. 3.

Proposed shadow-aware transformer encoder.

Diffusion-inspired refinement module (DIRM)

The DIRM delivers precise shadow removal by emulating the iterative denoising process of denoising diffusion probabilistic models while remaining computationally lightweight. DIRM refines intermediate reconstructions by injecting small amounts of random noise into feature maps and applying a residual correction block , yielding progressive updates as shown in Eq. (11) which make only minor adjustments, preserving fine details around complex shadow boundaries and textured surfaces. This strategy stabilizes refinement by combining stochastic noise with residual learning, allowing the learnable step size to adapt the correction magnitude and ensure convergence in few iterations. Moreover, unlike full denoising diffusion probabilistic models pipelines that require multiple costly forward–reverse passes, DIRM performs a small number of inexpensive residual convolutions per frame, enabling real-time shadow removal while maintaining high-fidelity, shadow-free results. The proposed refinement mechanism is entirely learning-based and does not rely on classical optimization techniques such as total variation minimization, bilateral filtering, or guided filtering. All smoothing and boundary preservation effects are learned end-to-end through the residual correction network.

11

where denotes the intermediate reconstruction at iteration , is a residual correction function parameterized by learnable weights, represents shadow priors generated by the encoder, and is a learnable step-size coefficient. The result is shadow-free pictures that look very real. DIRM takes ideas from diffusion models but avoids their difficult calculations. The residual convolution iterative model used in DIRM is illustrated in Fig. 4.

Fig. 4.

Proposed diffusion-inspired refinement module.

UNet decoder

The last part is a decoder network that uses skip connections from the MS-LSPM. These connections bring back the fine spatial details that were lost during encoding and global processing, so the decoder can produce high-resolution images with natural textures. It uses bilinear upsampling and simple convolutional refinement instead of heavy transposed convolutions, which keeps it efficient enough to be used on devices and in real-time applications such as robotics and self-driving cars.

From a system point of view, ShadowFormer++ balances well between performance and efficiency. Transformers and diffusion models are often resource-intensive. ShadowFormer++ achieves a balance by limiting the depth of the transformer and using a simple diffusion-inspired refinement. The tests show that this design can run in real time while outperforming other methods in terms of image quality across different datasets. In short, ShadowFormer++ is a step forward in shadow removal. By specifically modeling shadow priors, using prior-biased global attention, and using a refinement process inspired by denoising diffusion probabilistic models, it provides a solution that is robust, efficient, and adaptable. Figure 2 represents the architecture of the proposed framework.

Fig. 2.

Architecture of the proposed ShadowFormer++.

Training and inference of ShadowFormer++

The training and inference pipeline for ShadowFormer++ integrates multi-scale feature extraction, transformer-based analysis, and diffusion-based refinement. During training, shadow images undergo preprocessing via resizing and normalization. The MS-LSPM Module generates a hierarchical representation of shadow priors across multiple scales, capturing both global positioning and penumbra details. These priors are processed by the SATE, which utilizes global self-attention to establish dependencies between shadow and non-shadow regions. This mechanism ensures precise adjustments are maintained during complex scenarios. Subsequently, the DIRM progressively eliminates artifacts while preserving illumination consistency through a denoising diffusion process. The model is optimized using a combination of reconstruction loss, SSIM loss, and perceptual loss to ensure structural accuracy and visual fidelity. During inference, the system produces a shadow-free output from a single input image. Training hyperparameters are detailed in Table 1.

Table 1.

Hyperparameters for ShadowFormer++.

Parameter	Value	Description
Image resolution (Heightwidthchannels)	224 224 3	Fixed input image dimensions with RGB channels
Batch size	16	Number of samples processed per iteration
Learning rate	0.0001	Optimizer step size
Number of epochs	200	Total training iterations over the dataset
Patch size	16 16	Size of each image patch for tokenization
Number of tokens (N)	196	Total number of patch tokens
Embedding dimension (d)	384	Feature vector dimension for each token
Number of transformer layers	12	Depth of transformer encoder stack
Number of attention heads (h)	8	Parallel attention mechanisms per layer
Head dimension ()	48	Dimension per attention head (384/8)
MLP hidden dimension	1536	Intermediate feedforward layer size (4 384)
Layer normalization epsilon ()		Numerical stability constant for normalization
Shadow prior bias weight ()	0.1	Learnable scaling factor for shadow prior bias
MS-LSPM kernel sizes	33, 55, 77	Convolutional kernel sizes for MS-LSPM
Number of MS-LSPM branches	3	Number of parallel convolutional branches
Reconstruction loss weight ()	1.0	Weight for L1 pixel-wise reconstruction loss
SSIM loss weight ()	0.5	Weight for structural similarity loss
Perceptual loss weight ()	0.2	Weight for VGG-19 perceptual loss
Shadow prior loss weight ()	0.3	Weight for shadow prior regularization loss
Normalization range	[0.0, 1.0]	Min-max normalized pixel intensity range
Shadow mask values	{0.0, 1.0}	Binary mask values for shadow/non-shadow regions

Loss function

Loss functions quantify the difference between predicted and ground-truth outputs. So we employed a composite loss function, comprising a weighted sum of several distinct loss terms, to train ShadowFormer++. Specifically, our composite loss function includes reconstruction loss , structural similarity loss , perceptual loss , and shadow prior loss . This method represents a contemporary approach to model training, as reliance on a singular objective (such as pixel reconstruction) is generally insufficient for effective generalization. Given the complexity of shadow removal, employing a single metric does not adequately facilitate efficient model training. The primary objective is to balance perceptual realism, uniform illumination, structural coherence, and pixel quality of the images. We tackle this by utilizing diverse loss functions that ensure precision across multiple levels of representation.

Reconstruction loss ()

To ensure precise pixel-level accuracy, we use reconstruction loss, as it is less sensitive to large errors and tends to produce sharper images. It measures the absolute difference between the predicted shadow-free image and the ground truth on a per pixel, as shown in Eq. (12).

12

where N represents number of input images, is the predicted shadow-free image and the ground truth represented as .

Structural similarity loss ()

The SSIM assesses the perceived quality of an image by comparing luminance, contrast, and structure between two images. This loss aims to maximize the similarity and capture structural distortions that pixel-wise losses might not penalize. A model can have a good L1 score but still produce a blurry or structurally inconsistent image. This loss emphasizes structural consistency, promoting sharper edges and more coherent visual textures in both shadowed and non-shadowed regions. The calculation of Structural Similarity Loss is represented in Eq. (13).

13

where is the predicted shadow-free image and is the ground truth of the images.

Perceptual loss ()

To bridge the gap between low-level accuracy and high-level visual realism, we employ a perceptual loss based on feature matching from a pre-trained VGG-19 network, which a network trained on object recognition, it encourages the output to be perceptually and semantically similar to the ground truth. This is crucial for making the shadow-free result real and preventing the model from producing incorrect textures or artifacts. The perceptual loss is mathematically represented in Eq. (14).

14

Here, denotes the activation at the -th layer of the VGG-19 network, is the predicted shadow-free image and is the ground truth of the images, and , , are the channel, height, and width, respectively.

Shadow prior loss ()

To enforce alignment with the MS-LSPM Module, we include a regularization loss function that penalizes divergence between the learned prior and the soft shadow prediction . We used a Binary Cross-Entropy (BCE) loss that acts as a regularizer. This helps the network maintain shadow-awareness during the learning process, especially in regions with subtle transitions. This is represented in Eq. (15).

15

where denotes binary cross entropy, is the learned prior and the soft shadow prediction the soft shadow prediction.

Total loss

By combining these different losses, we get a well-rounded objective. The model learns from pixel-level details (reconstruction loss), structural information (SSIM loss), semantic understanding (Perceptual loss), and task-specific knowledge (Prior loss). This allows ShadowFormer++ to find a good balance between accurate results and realistic reconstructions. The total loss function used to train ShadowFormer++ combines the above components with tunable weights as shown in Eq. (16).

16

The hyperparameter through are empirically selected to balance the contribution of each loss term. In our implementation, , , , and yielded the optimal trade-off between MAE and perceptual fidelity. This multiobjective formulation enables ShadowFormer++ to recover shadow-free images with high perceptual fidelity, structural sharpness, and semantic consistency. To analyze the influence of the loss weighting coefficients, we conducted a sensitivity study independently varying (SSIM loss), (perceptual loss), and (shadow prior loss) while keeping the remaining parameters fixed. The evaluation was performed under identical training conditions, and PSNR, SSIM, and MAE were recorded. As shown in Table 2, a sensitivity analysis was conducted to examine the influence of the loss weighting coefficients –. Starting from the baseline configuration , each coefficient was varied individually while keeping the others fixed. Specifically, was varied to 0.3 and 0.7, to 0.1 and 0.4, and to 0.1 and 0.5. The results indicate that the baseline setting provides a balanced trade-off between reconstruction accuracy, structural consistency, and perceptual quality.

Table 2.

Sensitivity analysis of loss weighting coefficients.

Experiment					PSNR	SSIM	MAE
Proposed baseline	1	0.5	0.2	0.3	26.25	0.902	10.35
Lower SSIM (=0.3)	1	0.3	0.2	0.3	26.23	0.889	10.35
Higher SSIM (=0.7)	1	0.7	0.2	0.3	25.61	0.898	10.76
Lower prior (=0.1)	1	0.5	0.2	0.1	26.27	0.904	10.18
Higher prior (=0.5)	1	0.5	0.2	0.5	25.79	0.896	10.70
Lower perceptual (=0.1)	1	0.5	0.1	0.3	25.12	0.895	11.58
Higher perceptual (=0.4)	1	0.5	0.4	0.3	24.62	0.909	12.83

The Algorithm 1 outlines a systematic step-by-step procedure of the Shadow Removal Pipeline for ShadowFormer++. To summarize, the working of ShadowFormer++ image shadows are eliminated well by combining the MS-LSPM, the SATE, and the DIRM within a unified multibranch architecture. First, the input image is preprocessed by resizing and normalizing the pixel intensity. Then, the MS-LSPM generates explicit shadow priors to localize shadowed regions and preserve illumination awareness. After that, SATE uses these guides to focus its attention, telling the difference between illuminated and shadowed regions, and reasoning across long-range dependency. The DIRM refines images using a step-by-step correction method, similar to how diffusion improves models. A U-Net style decoder gives a coarse shadow-free reconstruction of the image. During training, the model gets better by using a mix of L1, SSIM, perceptual and shadow prior. When running the system, it only needs to go through these steps once to produce a high quality shadow free image.

Algorithm 1.

ShadowFormer++: shadow removal algorithm.

Experimental setup

In this section, we discussed the implementation details and the experimental setup used to execute our model, along with information about the datasets used for the experiments.

Dataset description

We conducted a series of shadow removal experiments using three benchmark datasets the ISTD Dataset³⁶, the ISTD+ (adjusted ISTD) Dataset³⁷, and the SRD Dataset³⁸. The ISTD Dataset³⁶ consists of a total of 1,870 triplet set of images which are divided into two parts that is 1,330 training sets and 540 testing sets. This triplet set of images consists of shadow images, shadow masks, and shadow-free images. The resolution of each image in this dataset is 640 480 pixels. The ISTD dataset³⁶ consists of 135 training scenes and 45 testing scenes, covering indoor and outdoor environments. In this dataset has colored image which is represented in RGB color space¹⁶. The ISTD+ dataset³⁶ is a enhanced dataset of the original ISTD dataset³⁷. This dataset provides a strong basis for our shadow removal framework testing and training. By reducing the inconsistency of illumination between shadow and shadow-free images through an image processing technique³⁹. The number of images in this dataset is same as the ISTD dataset³⁶ having the same resolution but enhanced RGB color space due to the image enhancement³⁹. Lastly,the SRD dataset³⁸ that we used consists of 3,088 image pairs out of which 2,680 image pairs are used for training and 408 image pairs are used for testing. Since this dataset does not have shadow mask we use the predicted masks from⁴⁰ for training and testing. Using these three publicly available dataset we have evaluate the performance of the state-of-the-art models and our Proposed model. The datasets use is summarized in Table 3.

Table 3.

Summary of the datasets used.

Feature	ISTD36	ISTD+37	SRD38
Image type	Triplet of real-world images (shadow image, shadow mask, shadow-free image)	Triplet of real-world images (shadow image, shadow mask, shadow-free image)	Pairs of synthetic images (shadow image, shadow-free image)
Key annotation	Binary shadow mask.	Refined binary shadow mask with corrected illumination	No shadow mask provided by default (use DHAN to generate shadow mask)
Total images	1,870 triplets.	1,870 triplets.	30,000+ pairs originally, with a standard split of 3,088 pairs
Data split	Training: 1,330 triplets; Validation: 200; Testing: 340.	Training: 1,330 triplets; Validation: 200; Testing: 340	Training: 2680 pairs; Validation: 204; Testing: 204

Implementation detail

We implemented ShadowFormer++ using PyTorch and simulated all experiments in a Google Colab Pro environment equipped with a NVIDIA Tesla T4 GPU (16 GB VRAM). The model was trained using three benchmark datasets: ISTD³⁶, ISTD+³⁷, and SRD³⁸, which together provide a diverse set of indoor and outdoor scenes with varying shadow conditions. All input images were resized to pixels for each color channel and normalized to the range before being passed into the network. Resized is performed for computational efficiency, but the fully convolutional architecture allows inference on images of arbitrary resolution (subject to hardware constraints), and since shadow removal depends primarily on illumination consistency and structural relationships rather than absolute pixel resolution, this resizing does not limit generalization to high-resolution imagery⁴¹.

The MS-LSPM was initialized with random weights, while the perceptual loss network utilized a VGG-19 model pre-trained on ImageNet. We used the Adam optimizer with , , and an initial learning rate of . To balance memory usage and computational efficiency, a batch size of eight was used during the training process. The model was trained using 200 epochs, each of which comprised full forward and backward passes over the training set.

Three common metrics were used to assess the model that are MAE, SSIM, and PSNR. Both over the entire image and over the shadow and non-shadow regions, metrics were calculated independently. Across all evaluation benchmarks, ShadowFormer++ demonstrated superior performance both quantitatively and qualitatively consistently outperformed state-of-the-art shadow removal methods. All experiments were conducted using benchmark dataset splits such as ISTD, ISTD+, and SRD. The baseline methods were evaluated following their original protocols using the dataset splits and publicly available pre-trained models when provided. Our proposed framework was trained on the same benchmark datasets without using additional external training data, ensuring a consistent comparison setting.

Result and discussion

In this section, we discussed the evaluation metrics that we used to assess the performance of state-of-the-art shadow removal models^{7,13,15–17,20,28–31,42}, as well as our proposed technique. Additionally, we analyzed the results of each model and conducted a comparative analysis. We also compared the computational performance of each of the models in this section.

Evaluation metrics

We use three evaluation metrics MAE, SSIM, and PSNR to gauge how well our proposed shadow removal model performs. According to earlier research^39,40,43,44, these metrics are used to evaluate the performance of shadow removal models. The MAE between the ground truth and the predicted shadow-free image is measured. To provide a single score, it computes the absolute difference between each predicted value and its matching ground truth value, then averages these absolute differences⁴⁵. Better shadow removal is indicated by lower MAE values. PSNR is used to assess the effectiveness of various methods in the RGB color space. Improved results are indicated by higher PSNR values. SSIM is used in addition to MAE and PSNR to assess the shadow removal performance. It is a metric that assesses how similar two images are by considering perceptual variations between images, and then treats image degradation as a perceived change in structural information. It assigns a score between -1 and 1, where 1 denotes perfect structural resemblance, based on measurements of brightness, contrast, and structure.⁴⁴.

Quantitative result analysis

We tested ShadowFormer++ using the ISTD³⁶, ISTD+³⁷, and SRD³⁸ datasets. We measured how well it performed using MAE, PSNR, and SSIM. For each set of data, we looked at how the model performed in areas with shadows, areas without shadows, and throughout the image. As shown in Table 4 the ISTD dataset³⁶, ShadowFormer++ performs better than the other models we compared to it. In shadowed areas, where it is harder to fix the image due to lighting problems, our model had the lowest MAE and the highest SSIM. This means that it rebuilt the shadowed parts well and did not distort the image too much. It performed well as it avoided color distortion and shapening the image better than older models such as DeshadowNet¹⁵ and DSC¹⁷, which often left behind some problems.In areas without shadows, our model preserved small details instead of over smoothing the image. It also had better PSNR scores than transformer-based models such as SpA-Former²⁰. When viewing the entire image, ShadowFormer++ removed the shadows well and kept the details. This suggests that using both general and specific information is beneficial.

Table 4.

Quantitative results on ISTD dataset.

Model	Shadow region			Non-shadow region			All Image
	MAE ()	PSNR ()	SSIM ()	MAE ()	PSNR ()	SSIM	MAE ()	PSNR ()	SSIM ()
ST-CGAN16	9.99	33.74	0.981	6.05	29.51	0.958	6.65	27.44	0.929
DeshadowNet15	8.45	34.12	0.987	5.22	32.40	0.978	5.67	30.50	0.959
DSC17	8.72	34.64	0.984	5.04	31.26	0.969	5.59	29.00	0.944
DHAN29	7.73	35.53	0.988	5.29	31.05	0.971	5.66	29.11	0.954
BMNet28	7.60	35.61	0.988	4.59	32.80	0.976	5.02	30.28	0.959
Mask-ShadowGAN13	10.30	32.85	0.973	6.10	29.90	0.951	6.70	27.90	0.932
Spa-Former20	7.40	36.15	0.989	4.45	32.95	0.977	4.75	30.50	0.961
DC-ShadowNet30	8.10	33.90	0.985	5.80	31.10	0.970	6.00	29.05	0.950
ShadowMaskFormer31	7.00	36.50	0.989	4.20	33.20	0.979	4.50	31.00	0.963
ShadowFormer7	6.00	37.80	0.992	3.50	34.10	0.987	4.00	32.50	0.967
DiffShadow28	6.50	36.90	0.990	4.00	33.50	0.980	4.30	31.20	0.960
CRFormer35	6.96	35.66	0.988	5.63	31.25	0.968	5.85	29.32	0.957
ShadowGAN-Former33	6.82	36.65	0.990	4.00	33.61	0.980	4.46	30.97	0.967
FieldNet1	5.83	38.67	0.991	3.54	34.40	0.987	4.07	32.76	0.967
ShadowFormer++ (ours)	5.47	39.24	0.993	3.28	35.12	0.989	3.79	33.58	0.972

Significant values are in bold.

As indicated in Table 5 the ISTD+ dataset³⁷, which includes more complex outdoor settings and shadows of varying darkness, ShadowFormer++ once again dominated the three measured metrics. It is worth noting that it improved the shadowed areas more than it did in the ISTD dataset¹⁶. This suggests that MS-LSPM and Diffusion-Inspired Refinement Module (DIRM) can handle tougher lighting differences. Other models such as DHAN²⁹and BMNet²⁸either made the fixed areas too dark or had color problems, but ShadowFormer++ made the colors change naturally. The results in areas without shadows show that our model did not make the good areas worse, which some GAN-based methods still do by adding noise. Looking at the entire image, ShadowFormer++ was better than previous models by 33.58 dB in PSNR, which shows that it does a good job of balancing both areas.

Table 5.

Quantitative results on ISTD+ dataset.

Model	Shadow region			Non-shadow region			All Image
	MAE ()	PSNR ()	SSIM ()	MAE ()	PSNR ()	SSIM	MAE ()	PSNR ()	SSIM ()
ST-CGAN16	13.4	33.74	0.981	7.7	29.51	0.958	8.7	27.44	0.929
DeshadowNet15	9.65	34.12	0.987	5.22	32.40	0.978	5.67	30.50	0.959
DSC17	8.72	34.64	0.984	5.04	31.26	0.969	5.59	29.00	0.944
DHAN29	7.73	35.53	0.988	5.29	31.05	0.971	5.66	29.11	0.954
BMNet28	5.60	35.61	0.988	2.50	32.80	0.976	3.00	30.28	0.959
Spa-Former20	6.80	36.15	0.989	3.20	32.95	0.977	3.80	30.50	0.961
DC-ShadowNet30	8.10	33.90	0.985	5.80	31.10	0.970	6.00	29.05	0.950
ShadowMaskFormer31	5.90	36.50	0.989	2.30	33.20	0.979	3.20	31.00	0.963
ShadowFormer7	4.70	37.80	0.992	1.90	34.10	0.987	2.60	32.50	0.967
DiffShadow28	5.50	36.90	0.990	2.40	33.50	0.980	3.00	31.20	0.960
CRFormer35	5.30	35.66	0.988	2.30	31.25	0.968	2.90	29.32	0.957
ShadowGAN-Former33	5.50	36.65	0.990	2.40	33.61	0.980	3.00	30.97	0.967
FieldNet1	4.90	38.67	0.991	2.10	34.40	0.987	2.60	32.76	0.967
ShadowFormer++ (ours)	4.50	39.24	0.993	1.90	35.12	0.989	2.30	33.58	0.972

Significant values are in bold.

Table 6 for the SRD dataset³⁸, which is especially hard with very clear scenes and different shadow shapes, ShadowFormer++ did the best job of all the models using all measurements. In shadowed areas, it had the lowest , which means that it made the fewest mistakes when rebuilding the image, even with large shadows. Unlike methods that use diffusion, which sometimes blur the image, our diffusion-inspired fixes kept the small details, such as textures on the ground or buildings. In areas without shadows, ShadowFormer++ did not fix too much and kept a high SSIM, which is important for making the images look real. When we looked at the whole image, our model did better than older CNN-based models such as DeshadowNet¹⁵ and also better than newer transformer-based methods such as ShadowFormer⁷ and SpA-Former²⁰. This shows that combining transformer-based thinking with information about shadows at different sizes is a good idea.

Table 6.

Quantitative results on SRD dataset.

Model	Shadow region			Non-shadow region			All Image
	MAE ()	PSNR ()	SSIM ()	MAE ()	PSNR ()	SSIM	MAE ()	PSNR ()	SSIM ()
ST-CGAN16	10.50	32.00	0.970	5.00	32.10	0.960	5.80	29.50	0.930
DeshadowNet15	9.10	31.80	0.980	4.60	32.50	0.970	5.20	29.50	0.950
DSC17	8.62	30.65	0.960	4.41	31.94	0.965	5.71	27.76	0.903
DHAN29	8.94	33.67	0.978	4.80	34.79	0.979	5.67	30.51	0.949
BMNet28	6.61	35.05	0.981	3.61	36.02	0.982	4.46	31.69	0.956
Spa-Former20	7.20	36.10	0.981	3.10	35.00	0.986	4.00	32.00	0.960
DC-ShadowNet30	8.30	34.00	0.975	5.00	32.00	0.968	5.40	30.10	0.921
ShadowMaskFormer31	6.50	36.50	0.982	3.00	36.50	0.988	4.00	32.50	0.965
ShadowFormer7	5.50	37.80	0.991	2.00	37.00	0.993	3.00	33.50	0.980
DiffShadow28	6.00	37.00	0.989	2.50	36.00	0.985	3.50	32.00	0.950
CRFormer35	6.96	35.04	0.981	3.13	36.29	0.986	4.18	32.23	0.958
ShadowGAN-Former33	6.58	35.25	0.981	3.55	36.12	0.981	4.40	31.73	0.956
FieldNet1	4.86	37.89	0.990	3.33	37.21	0.993	3.56	33.17	0.981
ShadowFormer++ (ours)	4.52	38.56	0.992	3.05	37.89	0.994	3.38	33.92	0.983

Significant values are in bold.

Overall, the results presented in Table 4, Table 5, and Table 6 demonstrate that ShadowFormer++ effectively restores shadowed regions while preserving the visual integrity of non-shadowed areas, resulting in superior overall image quality. These quantitative improvements highlight the effectiveness of the MS-LSPM, SATE and DIRM in addressing the complex challenges of shadow removal. Collectively, these findings establish ShadowFormer++ as a robust and reliable solution for real-world shadow removal applications.

Qualitative result analysis

In addition to the numbers, we took a close look at the ISTD³⁶, ISTD+³⁷, and SRD³⁸ datasets to see how well ShadowFormer++ works. Figures 5, 6 and 7 show examples from each dataset, with shadowed inputs, what other models did, and what our model predicts.

Fig. 5.

Qualitative comparison of shadow removal results for ISTD dataset.

Fig. 6.

Qualitative comparison of shadow removal results for ISTD+ dataset.

Fig. 7.

Qualitative comparison of shadow removal results for srd dataset.

In the ISTD dataset³⁶, older models, such as DeShadowNet¹⁵and DSC¹⁷ often leave bits of shadow behind, mostly where light and dark areas meet. Transformer models such as SpA-Former²⁰ fix some global issues, but still cannot get the edges right or blend colors well. But ShadowFormer++ makes sharper edges, gets colors right, and smoothly transitions between shadow and light. This shows that the MS-LSPM component effectively captures fine-grained details while ensuring global structural consistency. The Qualitative result of our proposed model on ISTD dataset¹⁶ is shown on Fig. 5.

In the ISTD+ dataset³⁷, which features complex outdoor environments and diverse surface textures, existing models such as DHAN²⁹ and BMNet²⁸ occasionally introduce artifacts and color inconsistencies. In contrast, our model ensures natural visual results with consistent color and balanced brightness. The DIRM part is key here because it slowly fixes shadowed areas such as in a diffusion process, so the fixed areas blend in with the textures around them. Also, if there were no shadows, the model does not alter those regions, unlike some GAN models that make the image worse. The Qualitative result of our proposed model on ISTD+ dataset is shown on Fig. 6.

In the SRD dataset³⁸, the pictures also show that ShadowFormer++ is better. Dealing with large, odd shadows in city and nature scenes is tough for other techniques. They often blur details or fail to remove shadows entirely. In contrast, our model corrects lighting while maintaining fine details such as ground patterns, leaves, and building textures. ShadowFormer++ also avoids common problems such as losing texture, which proves that mixing shadow-smart transformer encoding with different scales is a good idea. The Qualitative result of our proposed model on SRD dataset is shown on Fig. 7.

The images across all datasets show that ShadowFormer++ does a better job at removing shadows and keeping the scene realistic. We do not see any added artifacts, color issues, or lingering shadows, which proves the model is sound. These visual improvements align with the data in Tables 4, 5 and 6, showing that the model excels in different shadow conditions.

Computational analysis

The runtime complexity of shadow removal methods depends significantly on their architectural strategies for spatial and channel interactions. Traditional CNN-based approaches, such as DeShadowNet¹⁵, DSC¹⁷, and DC-ShadowNet³⁰, operate with a linear per-pixel cost of O(HWC), where H, W, and C denote the image height, width, and the number of feature channels, respectively. Dual-branch or generative architectures such as BMNet²⁸, ST-CGAN¹⁶, and Mask-ShadowGAN¹³ incur approximately double this cost, O(2HWC), due to parallel processing in subnetworks.

In contrast, transformer-based models with global self-attention, including SpA-Former²⁰, ShadowMaskFormer³¹, and CRFormer³⁵, exhibit quadratic complexity, , owing to the need to compute pairwise attention across all spatial locations. Diffusion models, such as Diff-Shadow¹⁸, further increase computational expense by multiplying the per-iteration complexity by the number of refinement steps T, resulting in O(THWC). ShadowFormer++ achieves subquadratic computational throughput by combining three main components:

• MS-LSPM : a lightweight encoder–decoder framework extracts multi-scale features in linear time per spatial location, projecting raw pixel inputs to a fixed feature width for efficient downstream processing. The time complexity is O(HW).

• SATE: instead of global attention, local self-attention windows of size (HW) are used, capturing crucial contextual interactions with subquadratic complexity. The time complexity is .

• DIRM: a single-pass denoising network refines the coarse output at multiple scales, avoiding the iterative computational burden of standard diffusion models. The time complexity is .

The total computational complexity of ShadowFormer++ is shown in equation (17) which places it between linear and quadratic complexity levels.

17

The channel dimension C is omitted from this analysis since each module processes fixed-width feature maps independently of the spatial resolution. This architectural design enables ShadowFormer++ to balance high-performance shadow removal with practical efficiency, making it suitable for real-time and large-scale deployments. The time complexity is shown in Table 7. Along with the complexity analysis, the empirical benchmarking was conducted for the evaluation of the performance of the suggested model. The ShadowFormer++ reaches an FPS of 47.32 at an image size of 224224. The average time taken for the inference of each image is 21.13 ms. The quantitative comparison of the suggested framework with the other approaches is provided in Table 7. The FPS for the baseline methods was obtained from their original publications. Although the FieldNet has higher FPS, the suggested approach has good performance in real-time inference and restoration.

Table 7.

Architecture, time complexity, and empirical runtime comparison of shadow removal methods.

Model	Architecture type	Time complexity	FPS	Time (s)
DeShadowNet15	CNN encoder–decoder	O(HWC)	8	0.125
DSC17	CNN + direction-aware context modules	O(HWC)	9	0.111
DHAN29	Multi-scale CNN + hierarchical attention	O(HWC)	9	0.117
BMNet28	Dual-branch CNN (boundary + mask)	O(2HWC)	16	0.062
SpA-Former20	CNN + transformer fusion (spatial attention)		20	0.050
DC-ShadowNet30	Dual-context CNN + attention	O(HWC)	12	0.083
ST-CGAN16	Dual GANs	O(2HWC)	10	0.100
Mask-ShadowGAN13	CycleGAN + mask generator	O(2HWC)	11	0.091
ShadowMaskFormer31	Transformer (Mask2Former variant)		24	0.042
ShadowGAN-Former33	GAN + transformer hybrid		22	0.045
ShadowFormer7	Transformer with windowed attention		28	0.036
Diff-Shadow18	Diffusion + UNet backbone (T steps)	O(THWC)	6	0.167
CRFormer35	Cross-region transformer encoder–decoder		26	0.038
FieldNet1	Lightweight CNN + attention gating	O(HWC)	66	0.015
ShadowFormer++ (ours)	Hybrid CNN-Transformer + DIRM		47	0.021

Scalability analysis

For efficiency and consistency with existing studies, images are resized to a fixed resolution during evaluation. To further analyze the scalability of the proposed model on higher-resolution inputs, additional experiments employ a patch-based inference strategy, where larger images are divided into smaller patches and processed independently. The model undergoes evaluation at the standard resized resolution of as well as at higher resolutions including the original ISTD resolution (), 720p, and 1080p. As summarized in Table 8, the proposed framework maintains stable performance as the resolution increases. The MAE slightly increases from 3.79 at to 3.81, 3.83, and 3.86 for , 720p, and 1080p, respectively. Similarly, PSNR gradually decreases from 33.58 dB to 33.05 dB, 32.91 dB, and 32.67 dB, while SSIM decreases marginally from 0.972 to 0.970, 0.968, and 0.965. As expected, computational time increases with image resolution, rising from 0.021 seconds per image at to 0.112 seconds, 0.247 seconds, and 0.511 seconds per image for , 720p, and 1080p, respectively. These results demonstrate that the proposed model scales effectively to higher resolutions while maintaining competitive restoration quality.

Table 8.

Scalability analysis of the proposed framework.

Resolution	MAE	PSNR	SSIM	Time (s)	FPS
224224 (resized)	3.79	33.58	0.972	0.021	47.32
Original ISTD (640480)	3.81	33.05	0.970	0.112	8.93
720p (1280720)	3.83	32.91	0.968	0.247	4.05
1080p (19201080)	3.86	32.67	0.965	0.511	1.96

Training and validation loss analysis

Figure 8 illustrates the training and validation loss curves for the ISTD³⁶, SRD³⁸, and ISTD+³⁷datasets over 200 epochs. Across all three datasets, ShadowFormer++ exhibited stable convergence, with the training loss consistently decreasing to an overall value of 0.37 and the validation loss closely following at 0.38. This minimal gap between training and validation losses reflects strong generalization capability and limited overfitting. The parallel decline in validation loss alongside the training curve further confirms that the model effectively learns both global illumination consistency and fine-grained shadow boundary restoration. Notably, these loss values are comparable to those achieved by other state-of-the-art models, demonstrating that ShadowFormer++ achieves similar generalization performance while offering improvements in shadow removal quality. The consistency of these trends across diverse datasets highlights the robustness of the proposed architecture and loss design.

Fig. 8.

Training and validation loss curves across three datasets.

Ablation study

To evaluate the contribution of each component in ShadowFormer++, an ablation study was conducted on the ISTD dataset by removing or modifying individual modules while keeping the rest of the architecture unchanged. The complete model achieves the best performance with 33.58 PSNR, 0.972 SSIM, and 3.79 MAE. Removing MS-LSPM, SATE, and DIRM degrades performance significantly, demonstrating their importance for multi-scale perception, contextual attention, and refinement. To further validate the design choice, the DIRM module was replaced with a ResNet block of comparable parameter size, which achieved 29.58 PSNR and 9.95 MAE, still inferior to the full model. The quantitative results are reported in Table 9, while qualitative comparisons are shown in Fig. 9.

Table 9.

Ablation study of the proposed ShadowFormer++ on the ISTD dataset.

Component	PSNR ()	SSIM ()	MAE ()	Total loss ()
ShadowFormer++ (proposed framework)	33.58	0.972	3.79	0.0436
w/o MS-LSPM	28.28	0.96	8.17	5.9177
w/o SATE	26.62	0.957	9.78	0.0921
w/o DIRM	27.62	0.958	8.92	0.0478
ResNet	29.58	0.956	9.95	0.0658

Fig. 9.

Qualitative ablation results of ShadowFormer++ showing the effect of removing different modules.

Comparative analysis

In this subsection, we compared a range of deep learning-based shadow removal models, including our proposed ShadowFormer++, focusing on their architectural choices, inherent limitations, and performance metrics such as MAE, PSNR, and SSIM. Table 10 illustrates the evolution of shadow removal techniques from early CNN-based approaches to more advanced hybrid transformer–diffusion architectures.

Table 10.

Comparative analysis of deep learning-based shadow removal models (ISTD dataset).

Model	Architectural design	Limitations	MAE ()	PSNR ()	SSIM ()	Loss Function Used
DeShadowNet15	CNN-based	Limited generalization; fails to handle complex lighting and soft shadows effectively	5.67	30.50	0.959	Reconstruction loss
ST-CGAN16	Conditional GAN	High computational cost; produces artifacts in penumbra and transition regions	6.65	27.44	0.929	Adversarial + L1
Mask-ShadowGAN13	GAN with mask guidance	Sensitive to inaccurate mask generation; inconsistent texture reconstruction.	6	30.97	0.967	Adversarial + mask loss
DC-ShadowNet30	Direction-aware CNN	Struggles with complex illumination variations; limited adaptability to unseen domains.	6.00	29.00	0.950	L1 + Attention loss
DSC17	Direction-aware spatial context	Context modeling weak in cluttered scenes; fails to capture global illumination cues.	5.59	29.00	0.944	L1 + contextual loss
DHAN29	Hybrid CNN with attention	Heavy architecture; not suitable for real-time deployment; sensitive to shadow density.	5.66	29.11	0.954	L1 + attention loss
BMNet28	Boundary-aware CNN	Requires paired data; limited generalization on diverse outdoor conditions	5.02	30.28	0.959	L1 + boundary loss
SpA-Former20	Transformer-based	High computational complexity; slower inference and limited fine detail recovery	4.75	30.50	0.961	L1 + transformer loss
ShadowMaskFormer31	Mask-augmented transformer	Sensitive to shadow shape variations; struggles with dynamic lighting transitions.	4.50	31.00	0.963	L1 + mask + transformer
Diff-Shadow18	Diffusion Model	Excellent reconstruction quality but very slow inference time; unsuitable for real-time use.	4.30	31.20	0.960	Diffusion loss + L1
ShadowFormer7	Transformer-based	High computation cost; lacks temporal coherence and detailed texture preservation.	4.00	32.50	0.967	L1 + transformer loss
ShadowGAN-Former33	GAN + transformer	Limited temporal coherence; sensitive to mask accuracy	4.46	30.97	0.967	Adversarial + transformer + mask
CRFormer35	Transformer-based	Moderate inference speed; some texture loss in complex shadows	5.85	29.32	0.957	L1 + transformer loss
FieldNet1	CNN + transformer hybrid	Moderate complexity; good generalization on outdoor scenes	4.07	32.75	0.967	L1 + perceptual loss
ShadowFormer++ (proposed)	Transformer + diffusion hybrid	Minimal; achieves real-time efficiency with high structural fidelity and illumination consistency	3.79	33.58	0.972	L1 + perceptual + SSIM + shadow prior loss

Significant values are in bold.

Initial CNN-based models, like DeShadowNet¹⁵and DSC¹⁷, had demonstrated limited generalization and struggled to handle diverse or complex lighting conditions, often failing in the presence of soft or dynamic shadows. The field then shifted toward GAN-based models, including ST-CGAN¹⁶ and Mask-ShadowGAN, which used adversarial learning to enhance image realism. However, these models still exhibited notable drawbacks, such as the introduction of artifacts, reliance on mask accuracy, and inconsistent texture reconstruction, especially in penumbra and transition zones.

Transformer-based approaches, such as SpA-Former²⁰, ShadowFormer⁷, and ShadowFormer³¹, took advantage of self-attention mechanisms to improve global context understanding and ensure better lighting consistency across the image. Despite these improvements, their increased model complexity and computational demands made them less suitable for real-time deployment, particularly on resource-limited devices. Meanwhile, diffusion-based models like Diff-Shadow¹⁸achieved the best reconstruction fidelity but suffered from prohibitively slow inference times due to their iterative denoising processes.

ShadowFormer++ overcame these limitations by integrating multi-scale shadow priors, a shadow-aware transformer encoder, and a lightweight diffusion-inspired refinement module. This combination enabled robust local feature extraction, global context preservation, and fine-grained restoration, without compromising computational efficiency. As a result, ShadowFormer++ achieved the best overall results, obtaining the lowest MAE (3.79), highest PSNR (39.24 dB), and SSIM (0.972) among all compared models. Importantly, it maintained real-time processing capability and structural stability, making it highly suitable for applications requiring both accuracy and speed under complex lighting scenarios. Additionally, the model’s balanced parameter count and efficient architecture facilitated deployment in environments with limited computational resources.

Conclusion

This study presented ShadowFormer++, a comprehensive framework for shadow removal that integrates the MS-LSPM, the SATE, and the DIRM to achieve high-fidelity image reconstruction. By effectively leveraging both global and local contextual information, ShadowFormer++ preserves overall image structure, restores fine-grained shadow boundaries, and maintains illumination consistency across challenging scenarios. Experimental results on benchmark datasets ISTD, ISTD+, and SRD demonstrated that ShadowFormer++ outperformed state-of-the-art shadow removal methods in terms of accuracy, adaptability, and visual quality. With a MAE of 3.79, PSNR of 33.58 dB, and a SSIM of 0.972, the model offers superior shadow removal performance while ensuring computational efficiency for real-time and resource-constrained environments. The architectural design facilitates robust handling of various shadow types, including both hard and soft shadows, and enables precise localization at shadow boundaries. Potential future enhancements include integrating a lightweight, jointly optimized shadow detection module to enable end-to-end learning, which could further improve boundary consistency while minimizing computational overhead. Such advancements would further position ShadowFormer++ as a practical solution for deployment in robotics, autonomous systems, and augmented reality applications, where both high accuracy and efficiency are required.

Author contributions

S.M: Conceptualization, Implementation, Writing – original draft, Writing – review & editing, Diagrams. S.K: Conceptualization, Writing – original draft, Writing – review & editing, and Supervision. R.S: Conceptualization, Writing – review & editing, and Supervision. All authors reviewed the manuscript.

Data availability

The ISTD dataset analysed during the current study is available in the Github repository, https://github.com/DeepInsight-PCALab/ST-CGAN. The ISTD+ dataset analysed during the current study is available in the Github repository, https://github.com/cvlab-stonybrook/SID?tab=readme-ov-file. The SRD dataset analysed during the current study is available in the Kaggle repository, https://www.kaggle.com/datasets/zalandafridi/srd-dataset.

Declarations

Competing interests

The authors declare no competing interests.