Taming diffusion models for image restoration: a review

Abstract

Diffusion models (DMs) have achieved remarkable progress in generative modelling, particularly in enhancing image quality to conform to human preferences. Recently, these models have also been applied to low-level computer vision for photo-realistic image restoration (IR) in tasks such as image denoising, deblurring and dehazing. In this review, we introduce key constructions in DMs and survey contemporary techniques that make use of DMs in solving general IR tasks. We also point out the main challenges and limitations of existing diffusion-based IR frameworks and provide potential directions for future work.

This article is part of the theme issue ‘Generative modelling meets Bayesian inference: a new paradigm for inverse problems’.

1. Introduction

Image restoration (IR) is a long-standing and challenging research topic in computer vision, which generally has two high-level aims: (i) recover high-quality (HQ) images from their degraded low-quality (LQ) counterparts, and (ii) eliminate undesired objects from specific scenes. The former includes tasks like image denoising [1] and deblurring [2], while the latter contains tasks like rain/haze/snow removal [3] and shadow removal [4]. Figure 1 showcases examples of these applications. To solve different IR problems, traditional methods require task-specific knowledge to model the degradation and perform restoration in the spatial or frequency domain, by combining classical signal processing algorithms [6–8] with specific image-degradation parameters [9]. More recently, numerous efforts have been made to train deep learning models on collected datasets to improve performance on different IR tasks [10]. Most of them directly train neural networks on sets of paired LQ–HQ images with a reconstruction objective (e.g. ${\ell }_1$ or ${\ell }_2$ distances) as typical in supervised learning. While effective, this approach tends to produce over-smooth results, particularly in textures [11]. Although this issue can be alleviated by including adversarial or perceptual losses [12], the training then typically becomes unstable and the results often contain undesired artefacts or are inconsistent with the input LQ images [11].

Figure 1.

Generally, there are two types of IR tasks: (1) Recover images from their degraded versions and (2) eliminate undesired objects from specific scenes. Here, all top rows are LQ input images and the bottom rows are the corresponding HQ images generated by a diffusion-based IR model [5]. As observed, applying DMs for IR can produce photo-realistic results in line with human perceptual preferences.

Recently, generative diffusion models (DMs) [13] have drawn increasing attention due to their stable training process and remarkable performance in producing realistic images and videos [14]. Inspired by them, numerous works have incorporated the diffusion process into various IR problems to obtain high-perceptual/photo-realistic results [15–18]. However, these methods exhibit considerable diversity and complexity across various domains and IR tasks, obscuring the shared foundations that are key to understanding and improving diffusion-based IR approaches. In light of this, our paper reviews the key concepts in DMs and then surveys trending techniques for applying them to IR tasks. More specifically, the fundamentals of DMs are introduced in §2, in which we further elucidate the score-based stochastic differential equations (Score-SDEs) and then show the connections between denoising diffusion probabilistic models (DDPMs) and Score-SDEs. In addition, the conditional diffusion models (CDMs) are elaborated such that we can learn to guide the image generation, which is key in adapting DMs for general IR tasks. Several diffusion-based IR frameworks are then summarized in §3. In particular, we can leverage CDMs for IR from different perspectives including DDPMs, Score-SDE and their connections. The connection even yields a training-free approach for non-blind IR, i.e. for tasks with known degradation parameters. Finally, we conclude the paper with a discussion of the remaining challenges and potential future work in §4.

2. Generative modelling with DMs

Generative DMs are a family of probabilistic models employing an iterative process (e.g. Markov chains) to transform the data distribution into a reference distribution. In the following, §2a describes a typical formulation of DMs: the DDPMs [13,19], followed by §2b which generalizes this to Score-SDEs for a more detailed analysis of the diffusion/reverse process. Finally, in §2c, we further show how to guide DMs for conditional generation, which is a key enabling technique for diffusion-based IR.

(a). DDPMs

Given a variable $x_0$ sampled from a data distribution $p_0(x)$ , DDPMs [13,19] are latent variable models consisting of two Markov chains: a forward/diffusion process $q(x_{1:T}|x_0)$ and a reverse process $p_{\theta }(x_{0:T})$ . The forward process transfers $x_0$ to a Gaussian distribution by sequentially injecting noise. For simplicity, we set $q_0(x)=p_0(x)$ such that the forward process starts from the data distribution. Then the reverse process learns to generate new data samples starting from the Gaussian noise. An overview of the DDPM is shown in figure 2. Below, we explain the forward and backward processes, and provide details on how the DDPMs are trained.

Figure 2.

DDPMs. The forward path transfers data to Gaussian noise, and the reverse path learns to generate data from noise along the actual time reversal of the forward process. Here, the reverse transition distribution $p_{\theta }(x_{t-1}|x_t)$ represents the model we aim to learn, and the conditional posterior $q(x_{t-1}|x_t,x_0)$ is a tractable Gaussian which serves as the target distribution the model wants to match as the $L_t$ term in equation (2.7).

(i). Forward diffusion process

The forward process perturbs data samples $x_0$ to noise $x_T$ . It can be characterized by a joint distribution encompassing all intermediate states, represented in the form

$$\begin{array}{lcr}& q(x_{1:T}|x_0)=\prod _{t=1}^{T}q(x_t|x_{t-1}),x_0\sim p_0(x).& \end{array}$$ (2.1)

Here, the transition kernel $q(x_t|x_{t-1})$ is a handcrafted Gaussian given by

$$\begin{array}{lcr}& q(x_t|x_{t-1})=\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I),& \end{array}$$ (2.2)

where ${\beta }_{1:T}\in (0,1)$ is the variance schedule: a set of pre-defined hyper-parameters that ensure the forward process (approximately) converges to a Gaussian distribution. Let ${\displaystyle {\alpha }_t=1-{\beta }_t}$ and ${\overline{\alpha }}_t=\prod _{s=1}^t{\alpha }_s$ , equation (2.2) then allows us to marginalize the joint distribution of equation (2.1) to

$$\begin{array}{lcr}& q(x_t|x_0)=\mathcal{N}(x_t;\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I).& \end{array}$$ (2.3)

We usually set ${\beta }_1<{\beta }_2<\cdots <{\beta }_T$ such that ${\alpha }_1>{\alpha }_2>\cdots >{\alpha }_T\approx 0$ and the terminal distribution $q(x_T)\approx \mathcal{N}(x_T;0,I)$ is thus a standard Gaussian, which allows us to generate new data points by reversing the diffusion process starting from sampled Gaussian noise. Moreover, it is important to note that posteriors along the forward process are tractable when conditioned on the original data sample $x_0$ , i.e. $q(x_{t-1}|x_t,x_0)$ is a tractable Gaussian [19]. This tractability enables the derivation of the DDPM training objective, which we will describe in §2a(iii).

(ii). Reverse process

In contrast, the reverse process learns to match the actual time reversal of the forward process, which is also a joint distribution modelled by $p_{\theta }(x_{0:T})$ as follows:

$$\begin{array}{lcr}& p_{\theta }(x_{0:T})=p(x_T)\prod _{t=1}^T{p}_{\theta }(x_{t-1}|x_t),x_T\sim \mathcal{N}(0,I).& \end{array}$$ (2.4)

In DDPMs, the transition kernel $p_{\theta }(x_{t-1}|x_t)$ is defined as a learnable Gaussian:

$$\begin{array}{lcr}& p_{\theta }(x_{t-1}|x_t)=\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t),{\mathit{\Sigma }}_{\theta }(x_t,t)I),& \end{array}$$ (2.5)

where ${\mu }_{\theta }$ and ${\mathit{\Sigma }}_{\theta }$ are the parameterized mean and variance, respectively. Learning the latent variable model of equation (2.5) is key to DDPMs since it substantially affects the quality of data sampling. That is, we have to adjust the parameters $\theta$ until the final sampled variable $x_0$ is close to that sampled from the real data distribution.

(iii). Training objective

To learn the reverse process, we usually minimize the variational bound on the negative log-likelihood which introduces the forward joint distribution of equation (2.1) in the objective $L$ as

$${\displaystyle {\displaystyle \begin{array}{r}{\mathbb{E}}_{q_0(x)}[-\log p_{\theta }(x_0)]\le \underset{\text{negative evidence lower bound }}{\underbrace{{\mathbb{E}}_{q(x_{0:T})}[-\log \frac{p_{\theta }(x_{0:T})}{q(x_{1:T}\mid x_0)}]}}={\mathbb{E}}_q[-\log p(x_T)-\sum _{t=1}^T\log \frac{p_{\theta }(x_{t-1}\mid x_t)}{q(x_t\mid x_{t-1})}].\end{array}}}$$ (2.6)

Here, $p(x_T)$ is a standard Gaussian, $p_{\theta }(x_{t-1}|x_t)$ is the reverse transition kernel in equation (2.5) that we want to learn, and $q(x_t|x_{t-1})$ is the forward transition kernel of equation (2.2). This objective can be further rewritten according to

$${\displaystyle {\displaystyle \begin{array}{r}L:={\mathbb{E}}_q[\underset{L_T}{\underbrace{D_{KL}(q(x_T\mid x_0)\mid \mid p(x_T))}}+\sum _{t=2}^T\underset{L_{t-1}}{\underbrace{D_{KL}(q(x_{t-1}\mid x_t,x_0)\mid \mid p_{\theta }(x_{t-1}\mid x_t))}}\underset{L_0}{\underbrace{-\log p_{\theta }(x_0\mid x_1)}}],\end{array}}}$$ (2.7)

where $L_T$ is called the prior matching term and contains no learnable parameters, $L_{t-1}$ is the posterior matching term and $L_0$ the data reconstruction term that maximizes the likelihood of $x_0$ . Sohl-Dickstein et al. [19] prove that the conditional posterior distribution in $L_{t-1}$ is a tractable Gaussian: $q(x_{t-1}|x_t,x_0)=\mathcal{N}(x_{t-1};{\stackrel{\mathrm{~}}{\mu }}_t(x_t,x_0),{\stackrel{\mathrm{~}}{\beta }}_{t}I)$ , where the mean and variance are

$${\displaystyle {\tilde{\mu }}_t(x_t,x_0)=\frac{\sqrt{{\alpha }_t}(1-{\overline{\alpha }}_{t-1})}{1-{\overline{\alpha }}_t}{x}_t+\frac{\sqrt{{\overline{\alpha }}_{t-1}}{\beta }_t}{1-{\overline{\alpha }}_t}{x}_0,\text{and}{\tilde{\beta }}_t=\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}{\beta }_t.}$$ (2.8)

All terms in ${\stackrel{\mathrm{~}}{\beta }}_t$ are known and thus the posterior variance in equation (2.5) can be non‐parametric, i.e. ${\mathit{\Sigma }}_{\theta }(x_t,t)={\stackrel{\mathrm{~}}{\beta }}_t$ , which does not depend on $x_t$ and allows us to only focus on learning the posterior mean ${\mu }_{\theta }(x_t,t)$ . Specifically, applying the reparameterization trick to $q(x_t|x_0)$ of equation (2.3) gives an estimate of the initial state: $x_0=\frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t-\sqrt{1-{\overline{\alpha }}_t}{ϵ}_t)$ , which can be substituted into equation (2.8) to obtain: ${\stackrel{\mathrm{~}}{\mu }}_t(x_t,x_0)=\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-{\overline{\alpha }}_t}}{ϵ}_t)$ . The noise ${ϵ}_t$ then can be learned using a neural network ${ϵ}_{\theta }(x_t,t)$ , and the parameterized distribution mean can be rewritten as

$$\begin{array}{lcr}& {\mu }_{\theta }(x_t,t)=\frac{1}{{\sqrt{\alpha }}_t}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-{\overline{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)).& \end{array}$$ (2.9)

The transition kernel $p_{\theta }(x_{t-1}|x_t)$ of equation (2.5) is finally updated according to the following:

$$\begin{array}{lcr}& p_{\theta }(x_{t-1}|x_t)=\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t),{\stackrel{\mathrm{~}}{\beta }}_{t}I),& \end{array}$$ (2.10)

where the variance ${\stackrel{\mathrm{~}}{\beta }}_t$ is predefined as in equation (2.8). Note that, $p_{\theta }(x_{t-1}|x_t)$ now matches the form of $q(x_{t-1}|x_t,x_0)=\mathcal{N}(x_{t-1};{\stackrel{\mathrm{~}}{\mu }}_t(x_t,x_0),{\stackrel{\mathrm{~}}{\beta }}_{t}I)$ , to minimize the KL term of $L_{t-1}$ in equation (2.7). Also note that DDPMs only need to learn the noise network ${ϵ}_{\theta }(x_t,t)$ , for which it is common to use a U-Net architecture with several self-attention layers [13]. The noise network ${ϵ}_{\theta }(x_t,t)$ takes an image $x_t$ and a time $t$ as input, and outputs a noise image of the same shape as $x_t$ . More specifically, the scalar time $t$ is encoded into vectors similar to positional embedding [20] and is combined with $x_t$ in the feature space for time-varying noise prediction.

Simplified objective

We now have known expressions for all components of the objective $L$ in equation (2.7). However, its current form is not ideal to use for model training since it requires $L_t$ to be computed at every timestep of the entire diffusion process, which is time-consuming and impractical. Fortunately, the prior matching term $L_T$ can be ignored since it does not contain any parameters. By substituting equations (2.8) and (2.9) into equation (2.7), we also find that the final expanded version of the posterior matching term $L_{t-1}$ ( $t\in \{2,\dots ,T\}$ ) and the data reconstruction term $L_0$ have similar forms,

$$\begin{array}{lcr}& L_{t-1}=\frac{{\beta }_t}{2{\alpha }_t(1-{\overline{\alpha }}_{t-1})}{\mathbb{E}}_{x_0,ϵ}[\Vert {ϵ}_t-{ϵ}_{\theta }(x_t,t){\Vert }^2]\mathrm{and}{L}_0=\frac{1}{2{\alpha }_1}{\mathbb{E}}_{x_0,ϵ}[\Vert {ϵ}_1-{ϵ}_{\theta }(x_1,1){\Vert }^2],& \end{array}$$ (2.11)

where ${\displaystyle {\mathbb{E}}_{x_0,ϵ}}$  denotes ${\mathbb{E}}_{x_0\sim q_0(x),ϵ\sim \mathcal{N}(0,I)}$ . By ignoring the weights outside the expectations in equation (2.11), the final training objective can therefore be obtained according to the following [13]:

$${\displaystyle \begin{array}{r}{L}_{\text{simple}}:={\mathbb{E}}_{x_0,t,ϵ}[\Vert {ϵ}_t-{ϵ}_{\theta }(x_t,t){\Vert }^2]={\mathbb{E}}_{x_0,t,ϵ}[\Vert {ϵ}_t-{ϵ}_{\theta }(\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}\cdot ϵ,t){\Vert }^2],\end{array}}$$ (2.12)

which essentially learns to match the predicted and real added noise for each training sample and thus is also called the noise matching loss. Compared to the original objective $L$ in equation (2.7), $L_{\text{simple}}$ is a re-weighted version that puts more focus on larger timesteps $t$ , which empirically has been shown to improve the training [13]. Once trained, the noise prediction network ${ϵ}_{\theta }(x_t,t)$ can be used to generate new data $x_0$ by running equation (2.10) starting from $x_T\sim \mathcal{N}(0,I)$ , i.e. by iterating

$$\begin{array}{lcr}& x_{t-1}={\mu }_{\theta }(x_t,t)+\sqrt{{\stackrel{\mathrm{~}}{\beta }}_t}ϵ\mathrm{where}{\mu }_{\theta }(x_t,t)=\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-{\overline{\alpha }}_t}}{ϵ}_{\theta }(x_t,t)),& \end{array}$$ (2.13)

as a parameterized data sampling process, similar to that in Langevin dynamics [21].

(b). Data perturbation and sampling with SDEs

We have shown how DDPM works for data perturbation and data generation. We can further generalize the DDPM to stochastic differential equations, namely, Score-SDE [22], where both the forward and reverse processes are in continuous-time state spaces. This generalization offers a deeper insight into the mathematics behind DMs that underlies the success of diffusion-based generative modelling. Figure 3 shows an overview of the Score-SDE approach.

Figure 3.

Data perturbation and sampling with SDEs. In contrast to DDPMs, the Score-SDE continuously perturbs the data to Gaussian noise using a forward SDE, $\mathrm{d}x=f(x,t)\mathrm{d}t+g(t)\mathrm{d}w$ , and then generates new samples by estimating the score ${\nabla }_x\log p_t(x)$ and simulating the corresponding reverse-time SDE.

(i). Data perturbation with forward SDEs

Here, we construct variables $\{x(t){\}}_{t\in [0,T]}$ for data perturbation in continuous time, which can be modelled as a forward SDE defined by

$$\begin{array}{lcr}& \mathrm{d}x=f(x,t)\mathrm{d}t+g(t)\mathrm{d}w,x(0)\sim p_0(x),& \end{array}$$ (2.14)

where $f(x,t)$ and $g(t)$ are called the drift and diffusion functions, respectively, and $w$ is a standard Wiener process (also known as Brownian motion). We use $p_t(x)$ to denote the marginal probability density of $x(t)$ , and use $p(x(t)|x(s))$ to denote the transition kernel from $x(s)$ to $x(t)$ . Moreover, we always design the SDE to drift to a fixed prior distribution (e.g. standard Gaussian), ensuring that $x(T)$ becomes independent of $p_0(x)$ and can be sampled individually.

(ii). Sampling with reverse-time SDEs

We can sample noise and reverse the forward SDE to generate new data close to that sampled from the real data distribution. Note that reversing equation (2.14) yields another diffusion process, i.e. a reverse-time SDE [23] in the form

$$\begin{array}{lcr}& \mathrm{d}x=[f(x,t)-g(t{)}^2{\nabla }_x\log p_t(x)]\mathrm{d}t+g(t)\mathrm{d}\hat{w},x(T)\sim p_T(x),& \end{array}$$ (2.15)

where $\hat{w}$ is a reverse-time Wiener process and ${\nabla }_x\log p_t(x)$ is called the score (or score function). The score ${\nabla }_x\log p_t(x)$ is the vector field of $x$ pointing to the directions in which the probability density function has the largest growth rate [21]. Once the score is known for all time $t$ , simulating equation (2.15) in time allows us to sample new data from noise.

Earlier work such as the score-based generative models [21] often learn the score using score matching [24]. However, score matching is computationally costly and only works for discrete times. Song et al. [22] propose a continuous-time version that optimizes the following:

$$\begin{array}{lcr}& {\mathbb{E}}_{t,x(0),x(t)}[\Vert s_{\theta }(x(t),t)-{\nabla }_{x(t)}\log p_t(x(t)|x(0)){\Vert }^2],& \end{array}$$ (2.16)

where $t$ is uniformly sampled over $[0,T]$ , $x(0)\sim p_0(x)$ , $x(t)\sim p_t(x(t)|x(0))$ , and $s_{\theta }(x(t),t)$ represents the score prediction network. This objective ensures that the optimal score network, denoted $s_{\theta }^{\ast }(x(t),t)$ , from equation (2.16) satisfies $s_{\theta }^{\ast }(x(t),t)={\nabla }_x\log p_t(x)$ almost surely [22,25].

(iii). Interpreting DDPM with the variance preserving SDE

Notably, extending DDPM to an infinite number of timesteps (i.e. continuous timesteps) leads to a special SDE which gives a more reliable interpretation of the diffusion process, and allows us to optimize the sampling with more efficient SDE/ordinary differential equation (ODE) solvers [22,26]. Specifically, recall the DDPM perturbation kernel $q(x_t|x_{t-1})$ of equation (2.2) and write it in the form

$${\displaystyle x_i=\sqrt{1-{\beta }_t}{x}_{i-1}+\sqrt{{\beta }_i}{ϵ}_{i-1},ϵ\sim \mathcal{N}(0,I)\text{and}i=1,\dots ,N,}$$ (2.17)

where $i$ is the discrete timestep. Let us define an auxiliary set $\{{\overline{\beta }}_i=N{\beta }_t{\}}_{i=1}^N$ and obtain

$$\begin{array}{lcr}& x_i=\sqrt{1-\frac{{\overline{\beta }}_i}{N}}{x}_{i-1}+\sqrt{\frac{{\overline{\beta }}_i}{N}}{ϵ}_{i-1}.& \end{array}$$ (2.18)

As a preparation to convert functions from discrete-time to continuous-time, let ${\displaystyle \beta (\frac{i}{N}):={\overline{\beta }}_i}$ , ${\displaystyle x(\frac{i}{N}):=x_i}$ and ${\displaystyle ϵ(\frac{i}{N}):={ϵ}_i}$ . We can now rewrite equation (2.18) with the difference $\mathit{\Delta }t=\frac{1}{N}$ and time ${\displaystyle t\in 0,\frac{1}{N},\dots ,\frac{N-1}{N}}$ as follows:

$${\displaystyle \begin{array}{}\text{(2.19)}& x(t+\Delta t)& =\sqrt{1-\beta (t+\Delta t)\Delta t}x(t)+\sqrt{\beta (t+\Delta t)\Delta t}ϵ(t)\text{(2.20)}& & \approx x(t)-\frac{1}{2}\beta (t)\Delta tx(t)+\sqrt{\beta (t+\Delta t)}\sqrt{\Delta t}ϵ(t)\text{(Taylor series)}\text{(2.21)}& & \approx x(t)-\frac{1}{2}\beta (t)\Delta tx(t)+\sqrt{\beta (t)}\sqrt{\Delta t}ϵ(t),\end{array}}$$

where the two approximate equalities hold when ${\displaystyle \mathrm{\Delta }t\to 0}$ . Then we convert ${\displaystyle \mathrm{\Delta }t}$ to $\mathrm{d}t$ , ${\displaystyle \sqrt{\mathrm{\Delta }t}ϵ(t)}$ to $\mathrm{d}w$ and obtain the following: $\mathrm{d}x=-\frac{1}{2}\beta (t)x\mathrm{d}t+\sqrt{\beta (t)}\mathrm{d}w$ , which is a typical mean-reverting SDE (also known as the Ornstein–Uhlenbeck process [27]) that drifts towards a stationary distribution, i.e. a standard Gaussian in this case. Song et al. [22] also name it the variance preserving SDE (VP-SDE) and further illustrate that DDPM’s marginal distribution $q(x_t|x_0)$ in equation (2.3) is a solution to the VP-SDE. Therefore, we can use either the diffusion reverse process (equation 2.13) or the reverse-time SDE (equation 2.15) to sample new data from noise with the same trained DDPM. In addition, the score ${\nabla }_x\log p_t(x)$ can be directly computed from the marginal distribution $q(x_t|x_0)$ in equation (2.3),

$$\begin{array}{lcr}& {\nabla }_{x_t}\log p_t(x_t)=-\frac{x_t-\sqrt{{\overline{\alpha }}_t}{x}_0}{(1-{\overline{\alpha }}_t)}=-\frac{{ϵ}_t}{\sqrt{1-{\overline{\alpha }}_t}},& \end{array}$$ (2.22)

where ${ϵ}_t$ is from the reparameterization trick and can be approximated using the noise prediction network ${ϵ}_{\theta }(x_t,t)$ . Equation (2.22) thus shows how we convert the DM to an SDE (i.e. obtain the score ${\nabla }_x\log p_t(x)$ from noise ${ϵ}_{\theta }(x_t,t)$ ). Then, numerous efficient SDE/ODE solvers can be used to optimize DMs, further bringing interpretability and faster sampling [22].

(c). CDMs

So far, we have learned how to sample data from different types of DMs. However, all of the above methods are concerned with unconditional generation, which is insufficient for IR where we want to sample HQ images conditioned on degraded LQ images. Therefore, we present the CDM below.

Let us keep the diffusion process $q(x_{1:T}|x_0)$ of equation (2.1) unchanged and reconstruct the reverse process in equation (2.4) with a condition $y$ , i.e. $p_{\theta }(x_{0:T}|y)=p(x_T|y){\prod }_{t=1}^T{p}_{\theta }(x_{t-1}|x_t,y)$ . The conditional reverse kernel can then be modelled as

$$\begin{array}{lrr}& p_{\theta ,\varphi }(x_{t-1}|x_t,y)=Z\cdot p_{\theta }(x_{t-1}|x_t)p_{\varphi }(y|x_{t-1}),& \end{array}$$ (2.23)

where $p_{\varphi }(y|x)$ is an additional network that predicts $y$ from $x$ , and $Z=p_{\varphi }(y|x_t{)}^{-1}$ can be treated as a constant since it does not depend on $x_{t-1}$ . This equation yields an adjusted mean for the posterior distribution of equation (2.10), given by [28]

$$\begin{array}{lcr}& {\hat{\mu }}_{\theta }(x_t,t,y)={\mu }_{\theta }(x_t,t)+\eta \cdot {\stackrel{\mathrm{~}}{\beta }}_t{\nabla }_{x_t}\log p_{\varphi }(y|x_t),& \end{array}$$ (2.24)

where $\eta$ is the gradient scale (also called the guidance scale). Equation (2.22) further gives the score of the joint distribution $p_t(x_t,y)$ :

$${\displaystyle \begin{array}{}\text{(2.25)}& {\mathrm{\nabla }}_{x_t}\log p_t(x_t,y)& ={\mathrm{\nabla }}_{x_t}\log p_t(x_t)+{\mathrm{\nabla }}_{x_t}\log p_t(y|x_t)\text{(2.26)}& & \approx -\frac{1}{\sqrt{1-{\overline{\alpha }}_t}}({ϵ}_{\theta }(x_t,t)-\sqrt{1-{\overline{\alpha }}_t}{\mathrm{\nabla }}_{x_t}\log p_{\varphi }(y|x_t)),\end{array}}$$

which provides a conditional noise predictor ${\widehat{ϵ}}_{\theta }$ with the following form [28]

$$\begin{array}{lcr}& {\widehat{ϵ}}_{\theta }(x_t,t,y)={ϵ}_{\theta }(x_t,t)-\eta \cdot \sqrt{1-{\overline{\alpha }}_t}{\nabla }_{x_t}\log p_{\varphi }(y|x_t).& \end{array}$$ (2.27)

The conditional sampling is performed as a regular DDPM by substituting this new noise predictor ${\widehat{ϵ}}_{\theta }(x_t,t,y)$ into the posterior mean of equation (2.9). The gradient scale $\eta$ controls the performance trade-off between image quality and fidelity, i.e. lower $\eta$ produces photo-realistic results, and higher $\eta$ yields better consistency with the condition.

(i). Conditional SDE

Similar to guided diffusion, we can also change the score function to control the reverse-time SDE conditioned on the variable $y$ , i.e. by replacing ${\nabla }_x\log p_t(x)$ with ${\nabla }_x\log p_t(x|y)$ in equation (2.15). Since $p_t(x|y)\propto p_t(x)p_t(y|x)$ , the conditional score can be decomposed as

$$\begin{array}{lcr}& {\nabla }_x\log p_t(x|y)={\nabla }_x\log p_t(x)+{\nabla }_x\log p_t(y|x),& \end{array}$$ (2.28)

which means that we can simulate the following reverse-time SDE for conditional generation:

$$\begin{array}{lcr}& \mathrm{d}x=[f(x,t)-g(t{)}^2({\nabla }_x\log p_t(x)+{\nabla }_x\log p_t(y|x))]\mathrm{d}t+g(t)\mathrm{d}\hat{w},& \end{array}$$ (2.29)

where $x(T)\sim p_T(x|y)$ . Song et al. [22] show that we can use a separate network to learn $p_t(y|x)$ (e.g. a time-dependent classifier if $y$ represents class labels), or estimate its log gradient ${\nabla }_x\log p_t(y|x)$ directly with heuristics and domain knowledge.

With these CDMs, we can sample images with specified labels (such as dog and cat) or, as the main topic of this paper, recover clean HQ images from corrupted LQ inputs.

3. DMs for IR

Diffusion-based IR can be considered a special case of CDMs with image conditioning. We first introduce the concept of image degradation, which is a process that transforms an HQ image $x$ into an LQ image $y$ characterized by undesired corruptions. The general image degradation process can be modelled as follows:

$$\begin{array}{lcr}& y=A(x)+n,& \end{array}$$ (3.1)

where $A$ denotes the degradation function and $n$ is additive noise. As the examples show in figure 1, degradation can manifest itself in various forms such as noise, blur, rain and haze. IR then aims to reverse this process to obtain a clean HQ image from the corrupted LQ counterpart $y$ .

IR is further decomposed into two distinct settings, blind and non-blind IR, depending on whether or not the degradation parameters $A$ and $n$ in equation (3.1) are known. Blind IR is the most general setting, in which no explicit knowledge of the degradation process is assumed. Blind IR methods instead utilize datasets of paired LQ–HQ images for supervised training of models. Non-blind IR methods, in contrast, assume access to $A$ and $n$ . This is an unrealistic assumption for many important real-world IR tasks, and thus limits non-blind methods to a subset of specific IR tasks such as bicubic downsampling, Gaussian blurring, colourization, or inpainting with a fixed mask. In the following, we first describe the most straightforward diffusion-based approach for general blind IR tasks in §3a. Representative non-blind diffusion-based approaches are then covered in §3b. Finally, §3c covers more recent methods for general blind IR.

(a). Conditional direct DM

The most straightforward approach for applying DMs to general IR tasks is to use the CDM with image guidance from §2c. In the IR context, the term $p_{\varphi }(y|x)$ in equation (2.27) represents the image degradation model which can be either a fixed operator with known parameters or a learnable neural network, depending on the task. It is also noted that strong guidance (large $\eta$ in equation 2.27) leads to good fidelity but visually LQ results (e.g. over-smooth images), while weak guidance (small $\eta$ ) has the opposite effect [28]. Now, let us consider the extreme case: how about decreasing $\eta$ to zero, i.e. no guidance? A simple observation from equation (2.27) is that with $\eta =0$ , the conditional noise predictor learns the unconditional noise predictor directly: ${\widehat{ϵ}}_{\theta }(x_t,t,y)={ϵ}_{\theta }(x_t,t)$ , and the objective for diffusion-based IR is given by

$$\begin{array}{lcr}& L_{\mathrm{cddm}}={\mathbb{E}}_{x_0,y,t,ϵ}[\Vert {ϵ}_t-{\widehat{ϵ}}_{\theta }(\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}\cdot ϵ,t,y){\Vert }^2].& \end{array}$$ (3.2)

We name this the conditional direct diffusion model (CDDM), which essentially follows the same training and sampling procedure as DDPM, except for the condition $y$ in the noise prediction as shown in figure 4. As a result, the generated image can be of very high visual quality (it looks realistic), but often has limited consistency with the original HQ image [16], as can be observed for the examples in the right-hand part of figure 4. Fortunately, some IR tasks, such as image super-resolution, colourization and inpainting, are highly ill-posed and can tolerate diverse predictions. CDDM can then be effectively applied to these tasks for photo-realistic IR.

Figure 4.

Left: Overview of CDDM on the face inpainting case. The only change compared to DDPM (figure 2) is the reverse transition model $p_{\theta }(x_{t-1}|x_t,y)$ , which involves the LQ image $y$ in sampling to generate the corresponding HQ image. Right: Two IR examples (image super-resolution and inpainting) performed under the CDDM framework. These results look realistic but are not consistent with the original image.

One typical method is SR3 [16], which employs CDDM with a few modifications for image super-resolution. To condition the model on the LQ image $y$ , SR3 up-samples $y$ to the target resolution so that $y$ can be concatenated with the intermediate state $x_t$ along the channel dimension. Subsequently, Palette [29] extends SR3 to general IR tasks including colourization, inpainting, uncropping and JPEG restoration. There is more work [17,30] employing the same ‘direct diffusion’ strategy but adopting different restoration pipelines and additional networks for task-specific model learning. More recently, Wang et al. [31] propose StableSR, which further adapts a large-scale pretrained DM (Stable Diffusion [14]) for IR, by tweaking the noise predictor with image conditioning in the same way as for CDDM.

(b). Training-free CDMs

The key to the success of CDDM in IR lies in learning the conditional noise predictor ${\widehat{ϵ}}_{\theta }(x_t,t,y)$ by optimizing equation (3.2) on a dataset of paired LQ–HQ images. Unfortunately, this means that ${\widehat{ϵ}}_{\theta }(x_t,t,y)$ needs to be re-trained to handle tasks which are not included in the current training data, even in the non-blind setting where the degradation parameters $A$ and $n$ in equation (3.1) are known. For non-blind IR, a training-free approach can instead be derived by directly incorporating the degradation function into a pretrained unconditional DM, such as a DDPM.

With known degradation parameters, the term $p(y|x)$ also becomes accessible: $p(y|x)=\mathcal{N}(A(x),{\sigma }_n^{2}I)$ , if the noise $n$ is Gaussian. Traditional IR approaches often solve this problem using maximum a posteriori (MAP) estimation [6], as follows:

$$\begin{array}{lcr}& \hat{x}=\arg \underset{x}{\min }\frac{1}{2{\sigma }_n^2}\Vert y-A(x){\Vert }^2+\lambda \mathcal{P}(x),& \end{array}$$ (3.3)

where $\mathcal{P}(x)$ is a prior term empirically chosen to characterize the prior knowledge of $x$ . Then, a natural idea is to incorporate a pretrained unconditional DDPM into $\mathcal{P}(x)$ as a powerful learned image prior. Specifically, recall the conditional score of equation (2.28) in the form

$$\begin{array}{lcr}& {\nabla }_{x_t}\log p_t(x_t|y)={\nabla }_{x_t}\log p_t(x_t)+{\nabla }_{x_t}\log p_t(y|x_t),& \end{array}$$ (3.4)

where $x_t$ matches the diffusion state in DDPM, and the unconditional score ${\nabla }_{x_t}\log p_t(x_t)$ can be obtained from equation (2.22) and approximated with DDPM’s noise predictor, as ${\nabla }_{x_t}\log p_t(x_t)\approx s_{\theta }(x_t,t)=-\frac{{ϵ}_{\theta }(x_t,t)}{\sqrt{1-{\overline{\alpha }}_t}}$ . However, computing $p_t(y|x_t)$ in equation (3.4) is difficult since there is no obvious relationship between $y$ and the state $x_t$ . Fortunately, with Gaussian noise $n\sim \mathcal{N}(0,{\sigma }_n^{2}I)$ , Chung et al. [32] propose an approximation for ${\nabla }_{x_t}\log p_t(y|x_t)$ at each timestep $t$ :

$$\begin{array}{lcr}& {\nabla }_{x_t}\log p_t(y|x_t)\approx {\nabla }_{x_t}\log p_t(y|{\hat{x}}_0),\mathrm{where}{\hat{x}}_0=\frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t+(1-{\overline{\alpha }}_t)s_{\theta }(x_t,t)).& \end{array}$$ (3.5)

This can be obtained via Tweedie’s formula [32–35]. The approximation above is motivated by a Dirac approximation ${\nabla }_{x_t}\log p(y|x_t)={\nabla }_{x_t}\log \int p(y|x_0)p(x_0|x_t)d{x}_0\approx {\nabla }_{x_t}\log p(y|{\hat{x}}_0(x_t))$ , where ${\hat{x}}_0(x_t)$ is any Monte Carlo sample of $p(x_0|x_t)$ . However, the approximation usually exhibits high variance since it uses a single Monte Carlo sample, while drawing more samples incurs more computations. Moreover, it is worth noting that $p_t(y|{\hat{x}}_0)$ is a tractable Gaussian: $p_t(y|{\hat{x}}_0)=\mathcal{N}(A({\hat{x}}_0),{\sigma }_n^{2}I)$ . Computing ${\nabla }_{x_t}\log p_t(y|{\hat{x}}_0)$ and substituting it for ${\nabla }_{x_t}\log p_t(y|x_t)$ in equation (3.4) thus gives the following:

$$\begin{array}{lcr}& {\nabla }_{x_t}\log p_t(x_t|y)\approx s_{\theta }(x_t,t)-\frac{1}{2{\sigma }_n^2}{\nabla }_{x_t}\Vert y-A({\hat{x}}_0){\Vert }^2.& \end{array}$$ (3.6)

We can then incorporate this approximation equation (3.6) into the sampling of a pretrained DDPM,

$${\displaystyle \begin{array}{}\text{(3.7)}& x_{t-1}& =\frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t+(1-{\alpha }_t){\mathrm{\nabla }}_{x_t}\log p_t(x_t|y))+\sqrt{{\tilde{\beta }}_t}ϵ\text{(3.8)}& & \approx \frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t+(1-{\alpha }_t)[s_{\theta }(x_t,t)-\frac{1}{2{\sigma }_n^2}{\mathrm{\nabla }}_{x_t}\Vert y-A({\hat{x}}_0){\Vert }^2])+\sqrt{{\tilde{\beta }}_t}ϵ\text{(3.9)}& & =\underset{\text{data consistency term}}{\underbrace{-\frac{1-{\alpha }_t}{2{\sigma }_n^2\sqrt{{\overline{\alpha }}_t}}{\mathrm{\nabla }}_{x_t}\Vert y-A({\hat{x}}_0){\Vert }^2}}+\underset{\text{diffusion term}}{\underbrace{\frac{1}{\sqrt{{\overline{\alpha }}_t}}(x_t+(1-{\alpha }_t)s_{\theta }(x_t,t))+\sqrt{{\tilde{\beta }}_t}ϵ}},\end{array}}$$

where the first line is derived from equations (2.13) and (2.22) with additional condition $y$ . Note that the diffusion term in equation (3.9) is actually an unconditional sampling step in DDPM, where $s_{\theta }$ is obtained from equation (2.22) as $s_{\theta }(x_t,t)=-\frac{{ϵ}_{\theta }(x_t,t)}{\sqrt{1-{\overline{\alpha }}_t}}$ . By letting $\rho =\frac{1-{\alpha }_t}{2{\sigma }_n^2\sqrt{{\overline{\alpha }}_t}}$ represent the step size of the data consistency term and simplifying the diffusion term, we then finally have

$$\begin{array}{lcr}& x_{t-1}=-\rho {\nabla }_{x_t}\Vert y-A({\hat{x}}_0){\Vert }^2+{\mu }_{\theta }(x_t,t)+\sqrt{{\stackrel{\mathrm{~}}{\beta }}_t}ϵ,& \end{array}$$ (3.10)

where ${\mu }_{\theta }$ and $\stackrel{\mathrm{~}}{\beta }$ are the posterior mean and variance of equation (2.10), respectively. This approach is called the diffusion posterior sampling (DPS) [32,36]. Note that equation (3.10) is conceptually similar to the MAP estimate in equation (3.3), with ${\nabla }_{x_t}\Vert y-A({\hat{x}}_0){\Vert }^2$ as the data consistency term and ${\mu }_{\theta }(x_t,t)+\sqrt{{\stackrel{\mathrm{~}}{\beta }}_t}ϵ$ being a diffusion-based image prior. When the degradation parameters of equation (3.1) are known, DPS thus utilizes this knowledge to guide the sampling process of a pretrained DDPM, encouraging generated images to be consistent with the LQ input image $y$ .

However, DPS does rely on the approximation in equation (3.5), for which the approximation error approaches zero only when the noise $n$ of $y$ has a high variance: ${\sigma }_n\to \infty$ . For the case where the LQ image is noiseless, $y=A(x)$ , we would prefer to introduce the approach from figure 5 where the unconditional generated state ${\hat{x}}_t$ is refined using the known degradation $A$ and the LQ image $y$ . More specifically, since the term ${\nabla }_{x_t}\log p_t(y|x_t)$ now is unattainable (or non-approximable), we instead apply the forward marginal transition of equation (2.3) also on the LQ image $y$ to obtain $y_t\sim q(y_t|y)=\mathcal{N}(y_t;\sqrt{{\overline{\alpha }}_t}y,(1-{\overline{\alpha }}_t)I)$ , which is an intermediate state between $y$ and Gaussian noise. Then, we impose data consistency by projecting ${\hat{x}}_t$ onto a conditional path as follows:

Figure 5.

Overview of the projection-based CDM. There are two paths for the HQ image $x$ and LQ image $y$ , generated from the same DM. At each reverse step $t$ , the sampling first leverages the pretrained DDPM for unconditional generation, i.e. $p_{\theta }({\hat{x}}_t|x_{t+1})$ , and then refines ${\hat{x}}_t$ to $x_t$ with functions $H$ and $b$ as $x_t=H({\hat{x}}_t)+b(y_t)$ , where $y_t$ is obtained by applying the forward marginal transition equation (2.3) on the LQ image as $y_t\sim q(y_t|y)$ .

$$\begin{array}{lcr}& x_t=H({\hat{x}}_t)+b(y_t),\mathrm{where}{\hat{x}}_t={\mu }_{\theta }(x_t,t)+\sqrt{{\stackrel{\mathrm{~}}{\beta }}_t}ϵ,& \end{array}$$ (3.11)

where $H$ and $b$ are functions derived from the known degradation $A$ . For computational efficiency, the two functions are typically assumed to be linear and tailored to specific IR tasks. This projection-based method is referred to as iterative latent variable refinement [37]. In addition, for linear degradation problems, we can refine the intermediate state $x_t$ by decomposing $A$ into partitions and then combine them with the LQ image $y$ in the reverse process [15], or optimize $x_t$ using the Bayesian framework directly [38]. These approaches are similar to the projection-based method but can be more computationally efficient.

Recently, another class of approaches for training-free CDMs have emerged which are based on Feynman–Kac models and sequential Monte Carlo (SMC) samplers [39–41]. At the core, they wrap the approximations for $p_t(y|x_t)$ in the proposals of an SMC sampler, so that the marginal distributions of the sampler anneals to the target conditional one. This approach is statistically exact, regardless of the approximations in $p_t(y|x_t)$ , in the sense that as the number of particles used in the SMC sampler goes to infinity, the resulting population converges in distribution to the target. As such, this type of method improves significantly over DPS [32] in terms of statistical errors. However, it comes at the cost of storing a population of particles which does not scale well in memory in the problem dimension, and the efficiencies of their proposals. To improve the sampler, Cardoso et al. [40] consider linear Gaussian likelihood models and propose efficient proposals based on an inpainting problem, while Janati et al. [41] develop a divide-and-conquer construction to set intermediate target distributions. Note that although Corenflos et al. [42], Dou & Song [43] also use SMC samplers, they target different Feynman–Kac models. Moreover, the methods in [42] are training-free only for special problems (e.g., inpainting).

(c). Diffusion process towards degraded images

In previous sections, we have presented several diffusion-based IR methods, both for the blind and non-blind settings. However, these methods all generate images starting from Gaussian noise, which intuitively should be inefficient for IR tasks, given that input LQ images are closely related to the corresponding HQ images. That is, it should be easier to translate directly from the LQ image to the HQ image, rather than from noise to HQ image, as shown in figure 6. To address this problem, for general blind IR tasks, Luo et al. [18] propose the IR-SDE that models image degradation with a mean-reverting SDE:

Figure 6.

Overview of the approach that performs diffusion towards degraded images. Here, the LQ image $y$ is involved in both the forward and backward processes. Moreover, the terminal state $x_T$ is a (noisy) LQ image rather than Gaussian noise.

$$\begin{array}{lcr}& dx={\theta }_t(\mu -x)dt+{\sigma }_{t}dw,& \end{array}$$ (3.12)

where $\mu$ is the state mean the SDE drifts to. The parameters ${\theta }_t$ and ${\sigma }_t$ are predefined and they control the speed of the mean-reversion and the stochastic volatility, respectively. It is noted that the VP-SDE [22] is a special case of equation (3.12) where $\mu$ is set to 0. Moreover, the SDE in equation (3.12) is proven to be tractable when the coefficients satisfy ${\sigma }_t^2/{\theta }_t=2{\lambda }^2$ for all timesteps [18]. Similar to DDPM, we can obtain the marginal transition kernel $p_t(x)$ , which is a Gaussian given by

$$\begin{array}{lcr}& p_t(x_t|x_0)=\mathcal{N}(x_t;\mu +(x_0-\mu ){\mathrm{e}}^{-{\overline{\theta }}_t},{\lambda }^2(1-{\mathrm{e}}^{-2{\overline{\theta }}_t})),& \end{array}$$ (3.13)

where ${\overline{\theta }}_t={\int }_0^t{\theta }_z\mathrm{d}z$ . As $t\to \infty$ , the terminal distribution converges to a stationary Gaussian with mean $\mu$ and variance ${\lambda }^2$ . By setting the HQ image as the initial state $x_0$ and the LQ image as the terminal state mean $\mu$ , this SDE iteratively transforms the HQ image into the LQ image with additional noise (where the noise level is fixed to $\lambda$ ). Then, we can restore the HQ image based on the reverse-time process of equation (3.12) as follows:

$${\displaystyle \mathrm{d}x=[{\theta }_t(\mu -x)-{\sigma }_t^2{\mathrm{\nabla }}_x\log p_t(x)]\mathrm{d}t+{\sigma }_t\mathrm{d}\hat{w}.}$$ (3.14)

Notably, the score function ${\nabla }_x\log p_t(x)$ is tractable when conditioning on the known $x_0$ in training, as ${\nabla }_x\log p_t(x|x_0)=-\frac{x_t-m_t}{v_t}$ , where $m_t$ and $v_t$ are the mean and variance of equation (3.13), respectively. Learning this score with a neural network is similar to denoising score matching [25] but the target score is directly computed from the training distributions.

However, IR-SDE still needs to add noise to the LQ image as a terminal state $x_T$ . For fixed point-to-point mapping with a diffusion process, we further introduce the diffusion bridge (DB) [44] which can naturally transfer complex data distributions to reference distributions, i.e. directly from HQ to LQ images, without adding noise. More specifically, given a diffusion process defined by a forward SDE as in equation (2.14), Rogers & Williams [45] show that we can force the SDE to drift from the HQ image $x$ to a particular condition (the LQ image $y$ ) via Doob’s $h$ -transform [46]:

$$\begin{array}{lcr}& \mathrm{d}x=f(x,t)\mathrm{d}t+{g(t)}^{2}h(x_t,t,y,T)+g(t)\mathrm{d}w,& \end{array}$$ (3.15)

where $h(x_t,t,y,T)={\nabla }_{x_t}\log p(x_T|x_t){|}_{x_T=y}$ is the gradient of the log transition kernel from $t$ to $T$ , derived from the original SDE. By setting the terminal state $x_T=y$ , the term ${g(t)}^{2}h(x_t,t,y,T)$ pushes each forward step towards the end condition $y$ , which exactly models the image degradation process. The corresponding reverse-time SDE of equation (3.15) can then be written as

$$\begin{array}{lcr}& \mathrm{d}x=[f(x,t)-g(t{)}^2(s(x_t,t,y,T)-h(x_t,t,y,T))]\mathrm{d}t+g(t)\mathrm{d}\hat{w},& \end{array}$$ (3.16)

where $s(x_t,t,y,T)={\nabla }_{x_t}\log p(x_t|x_T){|}_{x_T=y}$ is the conditional score function which can be learned via score-matching. The HQ image can then be recovered from the LQ image $y$ by iteratively running equation (3.16) in time as a traditional SDE solver. Note that we can design specific SDEs (e.g., VP/VE-SDE [22] to make the function $h(x_t,t,y,T)$ tractable [44,47,48]. The simplest case is the Brownian bridge [44] which constructs the marginal distribution as $p(x_t|x_0,x_T)=\mathcal{N}((1-\frac{t}{T})x_0+\frac{t}{T}{x}_T,\frac{t(T-t)}{T}I)$ . Another particular case is the Schrödinger bridge [48], which aims to compute a diffusion process that interpolates within the optimal coupling (when the reference measure of the bridge is chosen to be a Brownian motion) between the HQ and LQ image distributions [49]. The solution of the Schrödinger bridge converges weakly to an optimal transport plan with respect to 2-Wasserstein [48,50]. Most DB frameworks learn the noise ${ϵ}_{\theta }(x_t,t)$ directly by adopting the similar score reparameterization trick from equation (2.22), which leads to the following objective: $\Vert {ϵ}_{\theta }(x_t,t)-\frac{x_t-m_t}{\sqrt{v_t}}\Vert$ , where $m_t$ and $v_t$ are the marginal mean and variance of the forward process. More recently, Yue et al. [51] further propose to apply the DB to IR-SDE as the generalized Ornstein-Uhlenbeck bridge to achieve better performance. However, designing the forward SDE in equation (3.15) with a tractable yet effective $h(x_t,t,y,T)$ remains a challenge and is under-explored in IR. With the growing popularity of Score-SDEs and DBs, we hope that future approaches will offer various efficient and elegant solutions to general IR problems.

4. Conclusion and discussion

DMs have shown incredible capabilities and gained significant popularity in generative modelling. In particular, the mathematics behind them make these models exceedingly elegant. Building on their core concepts, we have described several approaches that effectively employ DMs for various IR tasks, achieving impressive results. However, it is also crucial to highlight the main challenges and further outline potential directions for future work.

(a). Difficult to process out-of-distribution degradations

Applying the trained DMs to out-of-distribution (OOD) data often leads to inferior performance and produces visually unpleasant artifacts [52], as shown in figure 7. There is research [31] proposing to address this issue by introducing the capable stable diffusion [14] with a feature control module [54]. However, such approaches still have to refine the stable DM with specific IR datasets. Moreover, the commonly used synthetic data strategy [55] just simulates known degradations such as noise, blur and compression and is unable to cover all corruption types that might be encountered in real-world applications. Inspired by the success of large language models and vision-language models, more recent approaches [5,52,53] have begun to explore the use of various language-based image representations in IR. The main idea is to produce ‘clean’ text descriptions of input LQ images, describing the main image content without undesired degradation-related concepts, and use these to guide the restoration process.

Figure 7.

Failed examples of applying a trained DM [53] to real-world and OOD LQ input images. In the left-hand example, the predicted HQ image contains unrecognizable text. In the right-hand example, the generated window shutters are visually unpleasant and inconsistent with the LQ input image.

(b). Inconsistency in image generation

While DMs produce photo-realistic results, the generated details are often inconsistent with the original input, especially regarding texture and text information, as shown in the right-hand part of figure 4 and in figure 7. This is mainly due to the intrinsic bias in the multi-step noise/score estimation and the stochasticity of the noise injection in each iteration. One solution is to add a predictor to generate the initial HQ image (with ${\ell }_1$ loss) and then gradually add more details via a diffusion process [30]. However, this requires an additional network and the performance highly depends on the trained predictor. IR-SDE [18] proposes a maximum likelihood objective to learn the optimal restoration path, but its reverse-time process still contains noise injection (i.e. Wiener process) thus leading to unsatisfactory results. Recently, flow matching and optimal transport have shown great potential in image generation. In particular, they can form straight‐line trajectories in inference, which are more efficient than the curved paths from the DMs [56]. The use of such methods for IR tasks is therefore a seemingly promising future direction.

(c). High computational cost and inference time

Most diffusion-based IR methods require a significant number of diffusion steps to generate the final HQ image (typically 1000 steps using DDPMs), which is both time-consuming and computationally costly, thus bringing challenges for deployment in various real-world applications. This problem can be alleviated using latent DMs [14,57] or efficient sampling techniques [26,58]. Unfortunately, these are not always suitable for IR tasks since the latent DM often produces colour shifting [31], and the efficient sampling would decrease the image generation quality [58]. Considering the particularity of IR, several works [18,42,48] design the diffusion process towards degraded images (§3c), such that their inference can start from the LQ image (rather than Gaussian noise). While this makes the sampling process more efficient (typically requiring less than 100 diffusion steps), it could be possible to improve further by designing more effective SDEs or DB functions.

(i). Closing

We have covered the basics of DMs and key techniques for applying them to IR tasks. This is an active research area with many interesting challenges and potential future directions, such as achieving photo-realistic yet consistent image generation, robustness to real-world image degradations, and more computationally efficient sampling. We hope that this review paper offers a foundational understanding that enables readers to gain deeper insights into the mathematical principles underlying advanced diffusion-based IR approaches.

Data accessibility

This article has no additional data.

Declaration of AI use

We have not used AI-assisted technologies in creating this article.

Authors’ contributions

Z.L.: conceptualization, investigation, methodology, project administration, validation, writing—original draft, writing—review and editing; F.G.: supervision, writing—original draft, writing—review and editing; Z.Z.: investigation, methodology, writing—review and editing; J.S.: supervision, writing—review and editing; T.S.: conceptualization, funding acquisition, supervision, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This research was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, by the project Deep Probabilistic Regression – New Models and Learning Algorithms (contract number: 2021-04301) funded by the Swedish Research Council, and by the Kjell & Märta Beijer Foundation.