Adaptive Weighted Discriminator for Training Generative Adversarial Networks

Abstract

Generative adversarial network (GAN) has become one of the most important neural network models for classical unsupervised machine learning. A variety of discriminator loss functions have been developed to train GAN’s discriminators and they all have a common structure: a sum of real and fake losses that only depends on the actual and generated data respectively. One challenge associated with an equally weighted sum of two losses is that the training may benefit one loss but harm the other, which we show causes instability and mode collapse. In this paper, we introduce a new family of discriminator loss functions that adopts a weighted sum of real and fake parts, which we call adaptive weighted loss functions or aw-loss functions. Using the gradients of the real and fake parts of the loss, we can adaptively choose weights to train a discriminator in the direction that benefits the GAN’s stability. Our method can be potentially applied to any discriminator model with a loss that is a sum of the real and fake parts. For our experiments, SN-GAN, AutoGAN, and BigGAN are used. Experiments validated the effectiveness of our loss functions on unconditional and conditional image generation tasks, improving the baseline results by a significant margin on CIFAR-10, STL-10, and CIFAR-100 datasets in Inception Scores (IS) and Fréchet Inception Distance (FID) metrics.

1. Introduction

Generative Adversarial Network (GAN) [15] has become one of the most important neural network models for unsupervised machine learning. The origin of this idea lies in the combination of two neural networks, one generative and one discriminative, that work simultaneously. The task of the generator is to generate data of a given distribution, while the discriminator’s purpose is to try to recognize which data are created by the generative model and which are the original ones. While a variety of GAN models have been developed, many of them are prone to issues with training such as instability where model parameters might destabilize and not converge, mode collapse where the generative model produces a limited number of different samples, diminishing gradients where the generator gradient vanishes and training does not occur, and high sensitivity to hyperparameters.

In this paper, we focus on the discriminative model to rectify the issues of instability and mode collapse in training GAN. In the GAN architecture, the discriminator model takes samples from the original dataset and the output from the generator as input and tries to classify whether a particular element in those samples is real or fake data [15]. The discriminator updates its parameters by maximizing a discriminator loss function via backpropagation through the discriminator network. In many of the proposed models [15, 16, 30, 28], the discriminator loss function consists of two equally weighted parts: the “real part” that purely relies on the original dataset and the “fake part” that depends on the generator network and its output; for simplicity we will call them ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ for real and fake losses, respectively. For example, in the original GAN paper [15], the discriminator loss function ${\mathcal{L}}_D$ is written as

$${\mathcal{L}}_D={\mathcal{L}}_r+{\mathcal{L}}_f,$$ (1)

with ${\mathcal{L}}_r={\mathbb{E}}_{x~p_d}[\log D(x)]$ and ${\mathcal{L}}_f={\mathbb{E}}_{z~p_z}[\log (1-D\text{(}G\text{(}z\text{)))]}$, where D and G are the discriminative and generative models, respectively, p_d is the probability distribution of the real data, and p_z is the probability distribution of the generator parameter z.

The goal of the GAN discriminator training is to increase both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ so that the discriminator D(·) assigns high scores to real data and low scores to fake data. This is done in (1) by placing equal weights on ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ [15]. However, the training with ${\mathcal{L}}_D$ is not performed equally on ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$. Indeed, a gradient ascent training step along the $\nabla {\mathcal{L}}_D$ may decrease ${\mathcal{L}}_r$ (or ${\mathcal{L}}_f$), depending on the angle between $\nabla {\mathcal{L}}_D$ and $\nabla {\mathcal{L}}_r$ (or $\nabla {\mathcal{L}}_f$). For example, if we have a large obtuse angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$, which is the case in most training steps (see §5.1), training along the direction of $\nabla {\mathcal{L}}_D$ may potentially decrease either ${\mathcal{L}}_r$ or ${\mathcal{L}}_f$ by going in the opposite direction to $\nabla {\mathcal{L}}_r$ or $\nabla {\mathcal{L}}_f$ (see §3 and §5.2). We suggest that this reduction on the real loss may destabilize training and cause mode collapses. Specifically, if a generator is converging with its generated samples close to the data distribution (or a particular mode), a training step that increases the fake loss will reduce the discriminator scores on the fake data and, by the continuity of D(·), reduce the scores on the nearby real data as well. With the updated discriminator now assigning lower scores to the regions of data where the generator previously approximated well, the generator update is likely to move away from that region and to the regions with higher discriminator scores (possibly a different mode). Hence, we see instability or mode collapse. See §5.3 for experimental results.

We propose a new approach in training the discriminative model by modifying the discriminator loss function and introducing adaptive weights in the following way,

$${\mathcal{L}}_D^{aw}=w_r\cdot {\mathcal{L}}_r+w_f\cdot {\mathcal{L}}_f.$$ (2)

We adaptively choose w_r and w_f weights to calibrate the training in the real and fake losses. Using the information of $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$, we can control the gradient direction, $\nabla {\mathcal{L}}_D^{aw}$, by either training in the direction that benefits both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ or increasing one loss while not changing the other. This attempts to avoid a situation where training may benefit one loss but significantly harm the other. A more detailed mathematical approach is presented in §3.

Our proposed method can be applied to any GAN model with a discriminator loss function composed of two parts as in (1). For our experiments we have applied adaptive weights to the SN-GAN [34] and the AutoGAN [14] models, and to the SN-GAN [34] and the BigGAN [5] models for unconditional and conditional image generating tasks, respectively. We have achieved significant improvements on them for CIFAR-10, STL-10 and CIFAR-100 datasets in both Inception Scores (IS) and Fréchet Inception Distance (FID) metrics, see §4. Our code is available at Github.

Notation:

We use ⟨·,·⟩₂ to denote the Euclidean inner product, ||x||₂ the Euclidean 2-norm, and ${\angle }_2(x,y):=\arccos \left(\frac{{\langle x,y\rangle }_2}{\Vert x{\Vert }_2\Vert y{\Vert }_2}\right)$ the angle between vectors x and y.

2. Related Work

GAN was first proposed in [15] for creating generative models via simultaneous optimization of a discriminative and a generative model. The original GAN may suffer from vanishing gradients during training, non-convergence of the model(s), and mode collapse; see [6, 31, 33, 38, 39] for discussions. Several papers [1, 16, 28, 30] have addressed the issues of vanishing gradients by introducing new loss functions. The LSGAN proposed in [30] adopted the least squares loss function for the discriminator that relies on minimizing the Pearson χ² divergence, in contrast to the Jensen–Shannon divergence used in GAN. The WGAN model [1, 16] introduced another way to solve the problem of convergence and mode collapse by incorporating Wasserstein-1 distance into the loss function. As a result, WGAN has a loss function associated with image quality which improves learning stability and convergence. The hinge loss function introduced in [28, 43] achieved smaller error rates than cross-entropy, being stable against different regularization techniques, and having a low computational cost [12]. The models in [3, 26, 45] adopted a loss function called maximum mean discrepancy (MMD). A repulsive function to stabilize the MMD-GAN training was employed in [46], and the MMD loss function was weighted in [11] according to the contribution of data to the loss function. [37] presented a dual discriminator GAN that combines two discriminators in a weighted sum.

New loss functions are not the only way of improving GAN’s framework. DCGAN [38], one of the first and more significant improvements in the GAN architecture, was the incorporation of deep convolutional networks. The Progressive Growing GAN [21] was created based on [1, 16] with the main idea of progressively adding new layers of higher resolution during training, which helps to create highly realistic images. [14, 13, 42] developed neural architecture search methods to find an optimal neural network architecture to train GAN for a particular task.

There are many works dedicated to the conditional GAN, for example BigGAN [5] which utilized a model with a large number of parameters and larger batch sizes showing a significant benefit of scaling.

There are many works devoted to improving or analyzing GAN training. [33] trained the generator by optimizing a loss function unrolled from several training iterations of the discriminator training. SN-GAN [34], normalized the spectral norm of each weight to stabilize the training. Recent work [40] introduced stable rank normalization that simultaneously controls the Lipschitz constant and the stable rank of a layer. [27] developed an analysis to suggest that first-order approximations of the discriminator lead to instability and mode collapse. [36] proved local stability under the model that both the generator and the discriminator are updated simultaneously via gradient descent. [9] analyzed the stability of GANs through stationarity of the generator. [32] points out that absolute continuity is necessary for GAN training to converge. Relativistic GAN [20] addressed the observation that with generator training increasing the probability that fake data is real, the probability of real data being real would decrease. [2] proposed a method of re-weighting training samples to correct for mass shift between the transferred distributions in the domain transfer setup. [7] viewed GAN as an energy-based model and proposed an MCMC sampling based method.

3. Adaptive Weighted Discriminator

If we maximize ${\mathcal{L}}_D$ to convergence in training discriminator D, we should meet the goal to increase both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$. However, when we train with a gradient ascent step along $\nabla \mathcal{L}=\nabla {\mathcal{L}}_r+\nabla {\mathcal{L}}_f$, which may be dominated by either $\nabla {\mathcal{L}}_r$ or $\nabla {\mathcal{L}}_f$, the training may be done primarily on one of the losses, either ${\mathcal{L}}_r$ or ${\mathcal{L}}_f$. Consider a gradient ascent training iteration for ${\mathcal{L}}_D$,

$${\theta }_1\leftarrow {\theta }_0+\lambda \nabla {\mathcal{L}}_D,$$ (3)

where λ is a learning rate. Then using the Taylor Theorem, we can expand both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ about θ₀,

$$\begin{array}{l}\\ {\mathcal{L}}_r\left({\theta }_1\right)={\mathcal{L}}_r\left({\theta }_0\right)+\lambda \nabla {\mathcal{L}}_r^T\nabla {\mathcal{L}}_D+\mathcal{O}\left({\lambda }^2\right)\\ ={\mathcal{L}}_r\left({\theta }_0\right)\end{array}$$ (4)

$$\begin{array}{l}+\lambda {\Vert \nabla {\mathcal{L}}_r\Vert }_2{\Vert \nabla {\mathcal{L}}_D\Vert }_2\cos \left({\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_D\right)\right)\\ +\mathcal{O}\left({\lambda }^2\right)\end{array}$$ (5)

and

$$\begin{array}{l}{\mathcal{L}}_f\left({\theta }_1\right)={\mathcal{L}}_f\left({\theta }_0\right)\\ +\lambda {\Vert \nabla {\mathcal{L}}_f\Vert }_2{\Vert \nabla {\mathcal{L}}_D\Vert }_2\cos \left({\angle }_2\left(\nabla {\mathcal{L}}_f,\nabla {\mathcal{L}}_D\right)\right)\\ +\mathcal{O}\left({\lambda }^2\right),\end{array}$$ (6)

where we have omitted the evaluation point θ₀ in all gradients (i.e.$\nabla {\mathcal{L}}_{*}=\nabla {\mathcal{L}}_{*}\left({\theta }_0\right)$) to avoid cumbersome expressions. If one of ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_D\right)$ and ${\angle }_2\left(\nabla {\mathcal{L}}_f,\nabla {\mathcal{L}}_D\right)$ is obtuse, then to the first order approximation, the corresponding loss is decreased. This causes a decrease in the discriminator assigning a correct score D(·) to the real (or fake) data. Thus, a gradient ascent step with loss (1) may turn out to decrease one of the losses if the angle ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\right)>{90}^{°}$. This situation occurs quite often in GAN training; see §5.1 for some experimental results illustrating this.

This undesirable situation is expected to happen in GAN training when the generator has produced samples close to the data distribution or its certain modes. If a training step in the direction $\nabla {\mathcal{L}}_D$ results in an increase in the fake loss or equivalently a decrease in the discriminator scores D(G(z)) on the fake data, it will decrease the scores D(x) on the real data as well by the continuity of D(·). Equivalently, this reduces the real loss. With the updated discriminator assigning lower scores to the regions of the data where the generator previously approximated well, the generator update using the new discriminator will likely move in the direction where the discriminator scores are higher and hence leave the region it was converging to. We suggest that this is one of the causes of instability in GAN training. If the regions with high discriminator scores contain only a few modes of the data distribution, this leads to mode collapse; see the study in §5.3.

To remedy this situation, we propose to modify the training gradient $\nabla {\mathcal{L}}_D$ to encourage high discriminator scores for real data. We propose a new family of discriminator loss functions, which we call adaptive weight loss function or aw-loss function; see equation (2).

We first show that the proposed weighted discriminator (2) with fixed weights carries the same theoretical properties of the original GAN as stated in [15, 32] for binary-cross-entropy loss function, i.e. when the min-max problem is solved exactly, we recover the data distribution.

Theorem 1.

Let p_d(x) and p_g(x) be the density functions for the data and model distributions, ${\mathbb{P}}_{\text{d}}$ and ${\mathbb{P}}_{\text{g}}$, respectively. Consider ${\mathcal{L}}^{aw}\left(D,p_{\text{g}}\right)=w_r{\mathbb{E}}_{x~p_d}[\log D(x)]+w_f{\mathbb{E}}_{x~p_g}[\log (1-D(x))]$ with fixed w_r, w_f > 0.

1. Given a fixed p_g(x), ${\mathcal{L}}^{aw}\left(D,p_{\text{g}}\right)$ is maximized by $D^{*}(x)=\frac{w_r{p}_{\text{d}}(x)}{w_r{p}_{\text{d}}(x)+w_f{p}_{\text{g}}(x)}$ for x ∈ supp(p_d) ∪ supp(p_g).

2. ${\min }_{p_{\text{g}}}{\max }_D{\mathcal{L}}^{aw}\left(D,p_{\text{g}}\right)=w_r\log \frac{w_r}{w_r+w_f}+w_f\log \frac{w_f}{w_r+w_f}$ with the minimum attained by p_g(x) = p_d(x).

See Appendix A for a proof of Theorem 1. To choose the weights w_r and w_f, we propose an adaptive scheme, where the weights w_r and w_f are determined using gradient information of both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$. This structure allows us to adjust the direction of the gradient of the discriminator loss function to achieve the goal of training to increase both ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$, or at least not to decrease either loss. We propose Algorithm 1 based on the following gradient relations with various weight choices.

Theorem 2.

Consider ${\mathcal{L}}_D^{aw}$ in (2) and the gradient $\nabla {\mathcal{L}}_D^{aw}$.

1. If $w_r=\frac{1}{{\Vert \nabla {\mathcal{L}}_r\Vert }_2}$ and $w_f=\frac{1}{{\Vert \nabla {\mathcal{L}}_f\Vert }_2}$, then $\nabla {\mathcal{L}}_D^{aw}$ is the angle bisector of $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$, i.e.

   $$\begin{array}{l}{\angle }_2\left(\nabla {\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_r\right)={\angle }_2\left(\nabla {\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_f\right)\\ ={\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\right)/2.\end{array}$$

2. If $w_r=\frac{1}{{\Vert \nabla {\mathcal{L}}_r\Vert }_2}$ and $w_f=-\frac{{\langle \nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\rangle }_2}{{\Vert \nabla {\mathcal{L}}_f\Vert }_2^2\cdot {\Vert \nabla {\mathcal{L}}_r\Vert }_2}$, then ${\angle }_2\left(\nabla {\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_f\right)={90}^{°}$,${\angle }_2\left(\nabla {\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_r\right)\le {90}^{°}$.

3. If $w_r=-\frac{{\langle \nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\rangle }_2}{{\Vert \nabla {\mathcal{L}}_r\Vert }_2^2\cdot {\Vert \nabla {\mathcal{L}}_f\Vert }_2}$ and $w_f=\frac{1}{{\Vert \nabla {\mathcal{L}}_f\Vert }_2}$, then ${\angle }_2\left({\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_r\right)={90}^{°}$,${\angle }_2\left(\nabla {\mathcal{L}}_D^{aw},\nabla {\mathcal{L}}_f\right)\le {90}^{°}$.

See Appendix A for a proof of Theorem 2. The first case in the above theorem allows us to choose weights for (2) such that we can train ${\mathcal{L}}_r$ and ${\mathcal{L}}_f$ by going in the direction of the angle bisector. However, sometimes the direction of the angle bisector might not be optimal. For example, if the angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$ is close to 180°, then the bisector direction will effectively not train either loss. During training, ${\mathcal{L}}_f$ is often easier to train than ${\mathcal{L}}_r$ meaning that the fake gradient has a larger magnitude. In this situation, we might want to train just on the real gradient direction by simply choosing wf = 0, but if the angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$ is obtuse, we will increase ${\mathcal{L}}_r$ but significantly decrease ${\mathcal{L}}_f$ which is undesirable. The second case in Theorem 2 suggests a direction that forms an acute angle with $\nabla {\mathcal{L}}_r$ and orthogonal to $\nabla {\mathcal{L}}_f$ (see Figure 1); such a direction will increase ${\mathcal{L}}_r$ and to the first order approximation will leave ${\mathcal{L}}_f$ unchanged. When ${\mathcal{L}}_r$ is high, the third case in Theorem 2 would allow us to increase ${\mathcal{L}}_f$ while minimizing changes to ${\mathcal{L}}_r$.

Figure 1:

Depiction of the second case of Theorem 2.

Inspired by the Theorem 2 and observations that we have made, we can calibrate discriminator training in a way that produces and maintains high real loss to reduce fluctuations in the real loss (or real discriminator scores) to improve stability. Algorithm 1 describes the procedure for updating weights of the aw-loss function in (2) during training using the information of $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$.

Algorithm 1 is designed to first avoid, up to the first order approximation, decreasing ${\mathcal{L}}_r$ or ${\mathcal{L}}_f$ during a gradient ascent iteration. Furthermore, it chooses to favor training real loss unless the mean real score is greater than the mean fake score (i.e. s_f ≤ s_r) and the real mean score is at least 0.5 (i.e. α₁ = 0.5 ≤ s_r). Here the mean discriminator scores s_r and s_f represent the mean probability that the discriminator assigns to x_i’s and y_j’s respectively as real data. When s_r is highly satisfactory with s_r ≥ α₂ = 0.75 (i.e. the midpoint between the minimum probability 0.5 and the maximum probability 1 for correct classifications of real data), we favor training the fake loss; otherwise, we train both equally. By maintaining these training criteria, we will reduce the fluctuations in real and fake discriminator scores and hence avoid instability. See study in §5.3. Note that we impose a small gap δ = 0.05 in s_f − δ > s_r to account for situations when s_r is nearly identical to s_f.

The way we favor training the real or fake loss depends on whether the angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$ is obtuse or not. In Algorithm 1, the first and the third cases are concerned with the more frequent situation (see §5.1 and Figure 4) where the angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$ is obtuse. These cases are the ones that are developed in Theorem 2. In the first case, we favor training real by going in the direction orthogonal to the $\nabla {\mathcal{L}}_f$, see Figure 1 for illustration. In the third case, we favor the fake loss by going in the direction orthogonal to $\nabla {\mathcal{L}}_r$. In a similar manner, the second and the fourth cases are concerned with the situation when the angle between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$ is acute. We use the same criteria to decide if training should favor the real or fake directions, but in this case we favor training the real or fake loss by using the direction of the corresponding gradient. Lastly, in the fifth case, it is desirable to increase both s_r and s_f without either taking priority, so we choose to train in the direction of the angle bisector between $\nabla {\mathcal{L}}_r$ and $\nabla {\mathcal{L}}_f$.

Figure 4:

Angles between gradients at each iteration. Top: original loss; Bottom: aw-loss.

The two threshold α₁ and α₂ in Algorithm 1 can be treated as hyperparameters. Our ablation studies show that the default α₁ = 0.5 and α₂ = 0.75 as discussed earlier are indeed good choices, see Appendix B.

All weights stated in Algorithm 1 normalize both real and fake gradients for the purpose of avoiding differently sized gradients, which has the effect of preventing exploding gradients and speeds up training, i.e. achieves better IS and FID with fewer epochs, see Figure 3. With this implementation, we implement a linear learning rate decay to ensure convergence. However, aw-method performs well without normalization, and achieves comparable results. We list the detailed results in Appendix B.

Figure 3:

AutoGAN vs aw-AutoGAN IS and FID plots for the first 40 epochs.

A small constant ε is added to all the weights in Algorithm 1 to avoid numerical discrepancies in cases that would prevent the discriminator model from training/updating. As an example, there are cases when our algorithm would set wr = 0 but at the same time $\nabla {\mathcal{L}}_f$ would be almost zero, which will result in $\nabla {\mathcal{L}}_D^{aw}$ being practically zero. We have set ε = 0.05 in all of our experiments.

Algorithm 1 has a small computational overhead. At each iteration we compute inner products and norms that are used for computing w_r and w_f, and then use these weight to update $\nabla {\mathcal{L}}_D^{aw}$. If we have k trainable parameters, then it takes an order of 6k operations to compute inner products between real–fake, real–real, and fake–fake gradients, and an order of 3k operations to form $\nabla {\mathcal{L}}_D^{aw}$, totalling to an order of 9k operations in Algorithm 1. This is a fraction of total computational complexity for one training iteration.

4. Experiments & Results

We implement our Adaptive Weighted Discriminator for SN-GAN [34] and AutoGAN [14] models, and for SN-GAN [34] and BigGAN [5] models, on unconditional and conditional image generating tasks, respectively (commonly referred to as unconditional and conditional GANs). AutoGAN is an architecture based on neural search. In our experiments and testings; we do not invoke a neural search with our aw-loss; we have simply implemented the aw-method on the model and architecture exactly provided by [14].

We test our method on three datasets: CIFAR-10 [25], STL-10 [10], and CIFAR-100 [25]. The datasets and implementation details provided in Appendix B.

All of the above mentioned models train the discriminator by minimizing the negative aw-hinge loss [28, 43]. Our aw-loss also uses the negative aw-hinge loss as follows:

$$\begin{array}{l}{\mathcal{L}}_D^{aw}=-w_r\cdot {\mathbb{E}}_{x~p_d}[\min (0,D(x)-1)]\\ -w_f\cdot {\mathbb{E}}_{z~p_z}[\min (0,-1-D(G(z))],\end{array}$$ (7)

with w_r and w_f updated every iteration using Algorithm 1.

To evaluate the performance of the models, we employ the widely used Inception Score [39] (IS) and Fréchet Inception Distance [18] (FID) metrics; see [29] for more details. We compute these metrics every 5 epochs and we report the best IS and FID achieved by each model within the 320 (SN-GAN), 300 (AutoGAN), and BigGAN (1,000) training epochs as in the original works.

We first present the results for three datasets CIFAR-10, STL-10m and CIFAR-100 in Tables 1, 2, and 3, respectively, for the unconditional GAN. In addition to baseline results, we have included top published results for each dataset for the comparison purposes.

Table 1:

CIFAR-10 (unconditional GAN): The aw-method substantially improves SN-GAN/AutoGAN.

Method	IS ↑	FID ↓
Imp. MMD GAN [46]	8.29	16.21
MGAN [19]	8.33±.10	26.7
MSGAN [44]	-	11.40
SRN-GAN [40]	8.53±.04	19.57
StyleGAN2 [22]	9.21±.09	8.32
SN-GAN [34]	8.22±.05	21.7
aw-SN-GAN (Ours)	8.53±.11	12.32
AutoGAN [14]	8.55±.10	12.42
aw-AutoGAN (Ours)	9.01±.03	11.82

Table 2:

STL-10 (unconditional GAN)

Method	IS ↑	FID ↓
ProbGAN [17]	8.87±.09	46.74
Imp. MMD GAN [46]	9.34	37.63
MGAN [19]	9.22±.11	-
MSGAN [44]	-	27.10
SN-GAN [34]	9.10±.04	40.10
aw-SN-GAN (Ours)	9.53±.10	36.41
AutoGAN [14]	9.16±.12	31.01
aw-AutoGAN (Ours)	9.41±.09	26.32

Table 3:

CIFAR-100 (unconditional GAN)

Method	IS ↑	FID ↓
SS-GAN [8]	-	21.02†
MSGAN [44]	-	19.74
SRN-GAN [40]	8.85	19.55
SN-GAN [34]	8.18±.12*	22.40*
aw-SN-GAN (Ours)	8.31±.02	19.08
AutoGAN [14]	8.54±.10*	19.98*
aw-AutoGAN (Ours)	8.90±.06	19.00

^*- results from our test

^†- quoted from [44].

For CIFAR-10 in Table 1, our methods significantly improve SN-GAN and AutoGAN baseline results. Indeed, our aw-AutoGAN achieves IS substantially above all comparisons other than StyleGAN2. Note that aw-AutoGAN uses 5.4M parameters vs. 26.2M for StyleGAN2.

For STL-10 in Table 2, our methods also significantly improve SN-GAN and AutoGAN baseline results. Our aw-SN-GAN achieved the highest IS and aw-AutoGAN achieved the lowest FID score among comparisons.

For CIFAR-100 in Table 3, our methods improve IS significantly for AutoGAN but modestly for SN-GAN. Our aw-Auto-GAN achieved the highest IS and the lowest FID score among comparisons.

We have also included some visual examples that were randomly generated by our aw-Auto-GAN model in Figure 2. We also consider the convergence of our method against training epochs by plotting in Figure 3 the IS and FID scores of 50,000 generated samples at every 5 epochs for AutoGAN vs aw-AutoGAN. For all the datasets, our model consistently achieves faster convergence than the baseline.

Figure 2:

aw-AutoGAN: CIFAR-10 (left), STL-10 (center), CIFAR-100 (right); samples randomly generated.

We next consider our aw-method for class conditional image generating task, using two base models, SN-GAN [34] and BigGAN [5], on CIFAR-10 and CIFAR-100 datasets. Results are listed in Table 4.

Table 4:

CIFAR-10 and CIFAR-100 (conditional GAN): The aw-method substantially improves SN-GAN and BigGAN.

	CIFAR-10		CIFAR-100
Method	IS ↑	FID ↓	IS ↑	FID ↓
cGAN [35]	8.62	17.5	9.04	23.2
CRGAN [47]	-	11.48	-	-
MHinge [23]	9.58±.09	7.50	14.36±.17	17.3
SNGAN [34]	8.60±.08	17.5	9.30†	15.6†
aw-SNGAN	9.03±.11	8.11	9.48±.13	14.42
BigGAN [5]	9.22	14.73	10.99±.14	11.73
aw-BigGAN [5]	9.52±.10	7.03	11.22±.17	10.23

^†- quoted from [41]; BigGAN [5] CIFAR-100 results based on our test using the code from [4].

Table 4 shows that our method works well for the conditional GAN too. The aw-method significantly improves the SN-GAN and BigGAN baselines.

5. Exploratory & Ablation Studies

In this section, we present three studies to illustrate potential problems of equally weighted GAN loss and advantages of our adaptive weighted loss. The hinge loss is implemented in the first and second studies, and a binary cross-entropy loss function is used for the third.

5.1. Angles between Gradients

In the first study, we examine the angles between $\nabla {\mathcal{L}}_r$, $\nabla {\mathcal{L}}_f$, $\nabla {\mathcal{L}}_D$ (or $\nabla {\mathcal{L}}_D^{aw}$). We use the CIFAR-10 dataset with the DCGAN architecture [38] and we look at 50 iterations in the first epoch of training. In Figure 4, we plot the following 3 angles:${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\right)$, ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_D\right)$ and ${\angle }_2\left(\nabla {\mathcal{L}}_f,\nabla {\mathcal{L}}_D\right)$ against iterations for the original loss ${\mathcal{L}}_D$ (1) on the top and for the aw-loss ${\mathcal{L}}_D^{aw}$ on the bottom. For the original loss (Left), ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_f\right)$(blue) stays greater then 90°, closer to 180°. ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_D\right)$ (green) often goes above 90° and so the training is often done to decrease the real loss. ${\angle }_2\left(\nabla {\mathcal{L}}_f,\nabla {\mathcal{L}}_D\right)$ also goes above 90°, though to a lesser extent. With the aw-loss (Right), ${\angle }_2\left(\nabla {\mathcal{L}}_r,\nabla {\mathcal{L}}_D^{aw}\right)$ and ${\angle }_2\left(\nabla {\mathcal{L}}_f,\nabla {\mathcal{L}}_D^{aw}\right)$ stay below the 90° line and indicate that we train in the direction of $\nabla {\mathcal{L}}_r$ and orthogonal to $\nabla {\mathcal{L}}_f$ in most steps.

5.2. Real Discriminator Scores and Real-Fake Gap after Training

Our second experiment is an ablation study to show that aw-loss increases the discriminator scores for real data and increases the gap between real and fake discriminator scores. We again apply the DCGAN model with the original loss ${\mathcal{L}}_D$ to CIFAR-10 and at every iteration we examine the mean discriminator score for the mini-batch of the real set and the mean discriminator scores for the mini-batch of the fake dataset generated by the generator. We use the logit output of the discriminator network as the score. We plot these two mean scores against each iteration before training in the first row of Figure 5 and after training (with the original loss ${\mathcal{L}}_D$) in the second row. At each of the above training iterations, we replace ${\mathcal{L}}_D$ by the aw-loss ${\mathcal{L}}_D^{aw}$ (2) and train for one iteration with the same training mini-batch. We plot the mean discriminator scores for the mini-batches of the real and fake dataset after this training in the third row of Figure 5. We further present the gaps between the two scores before training and after training using the original loss and using aw-loss in the fourth row of Figure 5.

Figure 5:

Mean discriminator scores for real data D(x) and fake data D(G(z)) (Row 1: before training, Row 2: after original training with ${\mathcal{L}}_D$, Row 3: after training with aw-loss ${\mathcal{L}}_D^{aw}$ and their gap (Row 4: GBT - gap before training; GAOT - gap after original training; GAAWT - gap after aw-loss training)

Figure 5 shows that training with aw-loss leads to higher real discriminator scores (0.921 epoch average) than training with the original loss (0.248 epoch average). The average gap between real and fake scores is also larger with the aw-loss at 1.413 vs. 1.262 of the original loss. Therefore, with the same model and the same training mini-batch, the aw-loss produces higher discriminator scores for the real dataset and larger gaps between real and fake scores. These are two important properties of a discriminator for the generator training.

5.3. Instability and Real Discriminator Scores

Our third study examines benefits of high discriminator scores for a real dataset with respect to instability and mode collapse of GAN training. We use a synthetic dataset with a mixture of Gaussian distributions used to test unrolled GAN in [33]. The dataset consists of eight 2D Gaussian distributions centered at eight equally distanced points on the unit circle. We train with a plain GAN as in [33] and we plot samples of (fake) data generated by the generator every 5,000 iterations on the first row of of Figure 6. We see that the generated data converging to two or three points but then moving off, demonstrating instability and mode collapse. To understand this phenomenon, at each of the iterations that we study in Figure 6, we generate 100 (real) data points from each of the eight classes and compute their mean discriminator scores (as the logit output of the discriminator). We plot the mean scores against the classes in the second row of Figure 6. We observe that the discriminator scores for the real data do not increase much during training, staying around 0, which corresponds to 0.5 probability after the logistic sigmoid function. The scores are also uneven among different classes. We believe these cause the instability in the generator training.

Figure 6:

Mixture of eight 2D Gaussian distributions centered at 8 points (right-most column). Row 1: GAN sample points produced by generators; Row 2: GAN mean discriminator scores for each of 8 classes; Row 3: aw-GAN sample points produced by generators; Row 4: aw-GAN mean discriminator scores for each of eight classes;

We compare the GAN results with aw-GAN that applies our adaptive weighted discriminator to the plain GAN and we present the corresponding plots of generated data points (fake) in the third row of Figure 6 and the corresponding discriminator scores on the eight classes in the bottom row. In this case, the generator gradually converges to all eight classes and the discriminator scores stay high for all eight classes. Even though the generator was starting to converge to a few classes (step 5,000), the discriminator scores remain high for all classes. Then the generator continues to converge while convergence to other classes occurs. We believe the high real discriminator scores maintains stability and prevents mode collapse in this case.

Conclusions

This paper pinpoints potential negative effects of the traditional GAN training on the real loss (and fake loss) and points out that this is a potential cause of instability and mode collapse. To remedy these issues, we have proposed the Adaptive Weighted Discriminator method to increase and maintain high real loss. Our experiments demonstrate the benefits and the competitiveness of this method.