Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping

Abstract

Unsupervised domain mapping aims to learn a function G_XY to translate domain $\mathcal{X}$ to $\mathcal{Y}$ in the absence of paired examples. Finding the optimal G_XY without paired data is an ill-posed problem, so appropriate constraints are required to obtain reasonable solutions. While some prominent constraints such as cycle consistency and distance preservation successfully constrain the solution space, they overlook the special properties of images that simple geometric transformations do not change the image’s semantic structure. Based on this special property, we develop a geometry-consistent generative adversarial network (Gc-GAN), which enables one-sided unsupervised domain mapping. GcGAN takes the original image and its counterpart image transformed by a predefined geometric transformation as inputs and generates two images in the new domain coupled with the corresponding geometry-consistency constraint. The geometry-consistency constraint reduces the space of possible solutions while keep the correct solutions in the search space. Quantitative and qualitative comparisons with the baseline (GAN alone) and the state-of-the-art methods including CycleGAN [66] and DistanceGAN [5] demonstrate the effectiveness of our method.

1. Introduction

Domain mapping or image-to-image translation, which targets at translating an image from one domain to another, has been intensively investigated over the past few years. Let $X\in \mathcal{X}$ denote a random variable representing source domain images and $Y\in \mathcal{Y}$ represent target domain images. According to whether we have access to a paired sample ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^N$, domain mapping can be studied in a supervised or unsupervised manner. While several works have successfully produced high-quality translations by focusing on supervised domain mapping with constraints provided by cross-domain image pairs [46, 26, 59, 58], the progress of unsupervised domain mapping is relatively slow.

In unsupervised domain mapping, the goal is to model the joint distribution P_XY given samples drawn from the marginal distributions P_X and P_Y in individual domains. Since the two marginal distributions can be inferred from an infinite set of possible joint distributions, it is difficult to guarantee that an individual input x ∈ X and the output G_XY(x) are paired up in a meaningful way without additional assumptions or constraints.

To address this problem, recent approaches have exploited the cycle-consistency assumption, i.e., a mapping G_XY and its inverse mapping G_YX should be bijections [66, 28, 61]. Specifically, when feeding an example x ∈ X into the networks G_XY ◦ G_YX : X → Y → X, the output should be a reconstruction of x and vise versa for y, i.e., G_YX (G_XY (x)) ≈ x and G_XY(G_YX(y)) ≈ y. Further, DistanceGAN [5] showed that maintaining the distances between images within domains allows one-sided unsupervised domain mapping.

Existing constraints overlook the special properties of images that simple geometric transformations (global geometric transformations without shape deformation) do not change the image’s semantic structure. Here, semantic structure refers to the information that distinguishes different object/staff classes, which can be easily perceived by humans regardless of trivial geometric transformations such as rotation. Based on this property, we develop a geometry-consistency constraint, which helps in reducing the search space of possible solutions while still keeping the correct set of solutions under consideration, and results in a geometry-consistent generative adversarial network (GcGAN).

Our geometry-consistency constraint is motivated by the fact that a given geometric transformation f(·) between the input images should be preserved by related translators G_XY and $G_{\tilde{X}\tilde{Y}}$, if $\tilde{\mathcal{X}}$ and $\tilde{\mathcal{Y}}$ are the domains obtained by applying f(·) on the examples of X and Y, respectively. Mathematically, given a random example x from the source domain $\mathcal{X}$ and a predefined geometric transformation function f(·), geometry consistency can be expressed as $f\left(G_{XY}(x)\right)\approx G_{\tilde{X}\tilde{Y}}(f(x))$ and $f^{-1}\left(G_{\tilde{X}\tilde{Y}}(f(x))\right)\approx G_{XY}\left(x\right)$, where f⁻¹(·) is the inverse function of f(·). Because it is unlikely that G_XY and $G_{\tilde{X}\tilde{Y}}$ always fail in the same location, G_XY and $G_{\tilde{X}\tilde{Y}}$ co-regularize each other by the geometry-consistency constraint and thus correct each others’ failures in local regions of their respective translations (see Figure 1 for an illustration). Our geometry-consistency constraint allows one-sided unsupervised domain mapping, i.e., G_XY can be trained independently from G_YX. In this paper, we employ two simple but representative geometric transformations as examples, i.e., vertical flipping (vf) and 90 degrees clockwise rotation (rot), to illustrate geometry consistency. Quantitative and qualitative comparisons with the baseline (GAN alone) and the state-of-the-art methods including CycleGAN [66] and Distance-GAN [5] demonstrate the effectiveness of our method.

Figure 1: Geometry consistency.

The original input image is denoted by x, and the predefined function f(·) is a 90° clockwise rotation (rot). GAN alone: $G_{XY}^1(x)$. GAN alone (rot): $f^{-1}\left(G_{\tilde{X}\tilde{Y}}^1(f(x))\right)$. GcGAN: $G_{XY}^2(x)$. GcGAN (rot): $f^{-1}\left(G_{\tilde{X}\tilde{Y}}^2(f(x))\right)$. It can be seen that GAN alone produces geometrically-inconsistent output images, indicating that the learned G_XY and $G_{\tilde{X}\tilde{Y}}$ are far away from the correct mapping functions. By enforcing geometry consistency, our method results in more sensible domain mapping.

2. Related Work

Generative Adversarial Networks.

Generative adversarial networks (GANs) [21, 45, 14, 47, 51, 3] learn two networks, i.e., a generator and a discriminator, in a staged zero-sum game fashion to generate images from inputs. Many tasks have recently been developed based on deep convolutional GANs, such as image inpainting, style transfer, and domain adaptation [7, 62, 46, 48, 31, 60, 9, 52, 23, 53, 64, 27, 50, 19, 18, 35, 63]. The key component enabling GANs is the adversarial constraint, which enforces the generated images to be indistinguishable from real images.

Domain Mapping.

Recent adversarial domain mapping has been studied in a supervised or unsupervised manner with respect to paired or unpaired inputs. There are a variety of literatures [46, 31, 26, 59, 56, 58, 25, 37, 4, 10] on supervised domain mapping. One representative example is conditional GAN [26], which learned the discriminator to distinguish (x, y) and (x, G_XY(x)) instead of y and G_XY(x), where (x, y) is a meaningful pair across domains. Further, Wang et al. [59] showed that conditional GANs can be used to generate high-resolution images with a novel feature matching loss, as well as multi-scale generator and discriminator architectures. While there has been significant progress in supervised domain mapping, many real-word applications can not provide aligned images across domains because data preparation is expensive. Thus, different constraints and frameworks have been proposed for image-to-image translation in the absence of training pairs.

In unsupervised domain mapping, only unaligned examples in individual domains are provided, making the task more practical but more difficult. Unpaired domain mapping has a long history, and some successes in adversarial networks have recently been presented [40, 66, 5, 39, 42, 38, 6, 11]. For example, Liu and Tuzel [40] introduced coupled GAN (CoGAN) to learn cross-domain representations by enforcing a weight-sharing constraint. Subsequently, CycleGAN [66], DiscoGAN [28], and DualGAN [61] enforced that translators G_XY and G_YX should be bijections. Thus, jointly learning G_XY and G_YX by enforcing cycle consistency can help to produce convincing mappings. Since then, many constraints and assumptions have been proposed to improve cycle consistency [8, 17, 24, 32, 34, 11, 2, 67, 20, 44, 39, 36, 1]. Recently, Benaim and Wolf [5] reported that maintaining the distances between samples within domains allows one-sided unsupervised domain mapping. GcGAN is also a one-sided framework coupled with our geometry-consistency constraint, and produces competitive and even better translations than the two-sided CycleGAN in various applications.

3. Preliminaries

Let $\mathcal{X}$ and $\mathcal{Y}$ be two domains with unpaired training examples ${\left\{x_i\right\}}_{i=1}^N$ and ${\left\{y_j\right\}}_{j=1}^M$, where x_i and y_j are drawn from the marginal distributions P_X and P_Y, where X and Y are two random variables associated with $\mathcal{X}$ and $\mathcal{Y}$, respectively. In the paper, we exploit style transfer without undesirable semantic distortions, and have two goals. First, we need to learn a mapping G_XY such that G_XY (X) has the same distribution as Y, i.e., $P_{G_{XY}(X)}\approx P_Y$. Second, the learned mapping function only changes the image style without distorting the semantic structures.

While many works have modeled the invertibility between G_XY and G_YX for convincing mappings since the success of CycleGAN, here we propose to enforce geometry consistency as a constraint that allows one-sided domain mapping. Let f(·) be a predefined geometric transformation. We can obtain two extra domains $\tilde{\mathcal{X}}$ and $\tilde{\mathcal{Y}}$ with examples ${\left\{{\tilde{x}}_i\right\}}_{i=1}^N$ and ${\left\{{\tilde{y}}_j\right\}}_{j=1}^M$ by applying f(·) on X and Y, respectively. We learn an additional translator $G_{\tilde{X}\tilde{Y}}:\tilde{X}\to \tilde{Y}$ while learning G_XY : X → Y, and introduce our geometry-consistency constraint based on the predefined transformation such that the two networks can co-regularize each other. Our framework enforces that G_XY(x) and $G_{\tilde{X}\tilde{Y}}(\tilde{x})$ should keep the same geometric transformation with the one between x and $\tilde{x}$, i.e., $f\left(G_{XY}(x)\right)\approx G_{\tilde{X}\tilde{Y}}(\tilde{x})$, where $\tilde{x}=f(x)$. We denote the two adversarial discriminators as D_Y and $D_{\tilde{Y}}$ with respect to domains $\mathcal{Y}$ and $\tilde{\mathcal{Y}}$_, respectively.

4. Proposed Method

We present our geometry-consistency constraint and Gc-GAN beginning with a review of the cycle-consistency constraint and the distance constraint. An illustration of the differences between these constraints is shown in Figure 2.

Figure 2: An illustration of the differences between CycleGAN [66], DistanceGAN [5], and our GcGAN.

x and y are random examples from domain $\mathcal{X}$ and $\mathcal{Y}$, respectively. d(x_i, x_j) is the distance between images x_i and x_j. f(·) is a predefined geometric transformation function for images, which satisfies $f^{-1}(f(x))=f\left(f^{-1}(x)\right)=x$. G_XY and $G_{\tilde{X}\tilde{Y}}$ are the generators (or translators) which target the domain translation tasks from $\mathcal{X}$ to $\mathcal{Y}$ and $\tilde{\mathcal{X}}$ to $\tilde{\mathcal{Y}}$, where $\tilde{\mathcal{X}}$ and $\tilde{\mathcal{Y}}$ are two domains obtained by applying f(·) on all the images in $\mathcal{X}$ and $\mathcal{Y}$, respectively. D_Y is an adversarial discriminator in domain $\mathcal{Y}$. The red dotted lines denote the unsupervised constraints with respect to cycle consistency (x ≈ G_YX (G_XY(x))), distance consistency (x ≈ G_YX (G_XY(x))), and our geometry consistency $\left(f\left(G_{XY}(x)\right)\approx G_{\tilde{X}\tilde{Y}}(f(x))\right)$, respectively.

4.1. Unsupervised Constraints

Cycle-consistency constraint.

Following the cycle-consistency assumption [28, 66, 61], through the translators G_XY ◦ G_YX : X → Y → X and G_YX ◦ G_XY : Y → X → Y, the examples x and y in domain $\mathcal{X}$ and $\mathcal{Y}$ should recover the original images, i.e., x ≈ G_YX (G_XY (x)) and y ≈ G_XY (G_YX (y)). Cycle consistency is implemented by a bidirectional reconstruction process that requires G_XY and G_YX to be jointly learned, as shown in Figure 2 (Cycle-GAN). The cycle consistency loss ${\mathcal{L}}_{cyc}\left(G_{XY},G_{YX},X,Y\right)$ takes the form as:

$$\begin{gathered} {\mathcal{L}}_{cyc}={\mathbb{E}}_{x~P_X}\left[\Vert G_{YX}\left(G_{XY}(x)\right)-x{\Vert }_1\right] \\ +{\mathbb{E}}_{y~P_Y}\left[\Vert G_{XY}\left(G_{YX}(y)\right)-y{\Vert }_1\right]. \end{gathered}$$ (1)

Distance constraint.

The assumption behind the distance constraint is that the distance between two examples x_i and x_j in domain X should be preserved after mapping to domain Y, i.e., d(x_i, x_j) ≈ a·d(G_XY (x_i), G_XY(x_j)) + b, where d(·) is a predefined function to measure the distance between two examples and a and b are the linear coefficient and bias. In DistanceGAN [5], the distance consistency loss ${\mathcal{L}}_{dis}\left(G_{XY},X,Y\right)$ is the exception to the absolute differences between distances:

$$\begin{gathered} {\mathcal{L}}_{dis}={\mathbb{E}}_{x_i,x_j~P_X}\left[\left|\varphi \left(x_i,x_j\right)-\psi \left(x_i,x_j\right)\right|\right], \\ \varphi \left(x_i,x_j\right)=\frac{1}{{\sigma }_X}\left(\Vert x_i-x_j{\Vert }_1-{\mu }_X\right), \\ \psi \left(x_i,x_j\right)=\frac{1}{{\sigma }_Y}\left(\Vert G_{XY}\left(x_i\right)-G_{XY}\left(x_j\right){\Vert }_1-{\mu }_Y\right), \end{gathered}$$ (2)

where μ_X, μ_Y (σ_X, σ_Y) are the means (standard deviations) of distances of all the possible pairs of (x_i, x_j) within domain $\mathcal{X}$ and (y_i, y_j) within domain $\mathcal{Y}$, respectively.

4.2. Geometry-consistent Generative Adversarial Networks

Adversarial constraint.

Taking G_XY as an example, an adversarial loss ${\mathcal{L}}_{gan}\left(G_{XY},D_Y,X,Y\right)$ [21] enforces G_XY and D_Y to simultaneously optimize each other in an minimax game, i.e., ${\text{min}}_{G_{XY}}{\text{max}}_{D_Y}{\mathcal{L}}_{gan}\left(G_{XY},D_Y,X,Y\right)$. In other words, D_Y aims to distinguish real examples {y} from translated samples {G_XY(x)}. By contrast, G_XY aims to fool D_Y so that D_Y can label a fake example $y^{\prime }=G_{XY}(x)$ as a sample satisfying $y^{\prime }~P_Y$. The objective can be expressed as:

$$\begin{gathered} {\mathcal{L}}_{gan}={\mathbb{E}}_{y~P_Y}\left[\text{log\hspace{0.17em}}{D}_Y(y)\right] \\ +{\mathbb{E}}_{x~P_X}\left[\text{log}\left(1-D_Y\left(G_{XY}(x)\right)\right)\right]. \end{gathered}$$ (3)

In the transformed domains $\tilde{\mathcal{X}}$ and $\tilde{\mathcal{Y}}$, we employ the adversarial loss ${\mathcal{L}}_{gan}\left(G_{\tilde{X}\tilde{Y}},D_{\tilde{Y}},\tilde{X},\tilde{Y}\right)$ that has the same form to ${\mathcal{L}}_{gan}\left(G_{XY},D_Y,X,Y\right)$.

Geometry-consistency constraint.

As shown in Figure 2 (GcGAN), given a predefined geometric transformation function f(·), we feed the images x ∈ X and $\tilde{x}=f(x)$ into the translators G_XY and $G_{\tilde{X}\tilde{Y}}$, respectively. Following our geometry-consistency constraint, the outputs $y^{\prime }=G_{XY}(x)$ and ${\tilde{y}}^{\prime }=G_{\tilde{X}\tilde{Y}}(\tilde{x})$ should also satisfy ${\tilde{y}}^{\prime }\approx f\left(y^{\prime }\right)$ like x and $\tilde{x}$. Considering both f(·) and the inverse geometric transformation function f⁻¹(·), our complete geometry consistency loss ${\mathcal{L}}_{geo}\left(G_{XY},G_{\tilde{X}\tilde{Y}},X,Y\right)$ has the following form:

$$\begin{gathered} {\mathcal{L}}_{geo}={\mathbb{E}}_{x~P_X}\left[\Vert G_{XY}(x)-f^{-1}\left(G_{\tilde{X}\tilde{Y}}(f(x))\right){\Vert }_1\right] \\ +{\mathbb{E}}_{x~P_X}\left[\Vert G_{\tilde{X}\tilde{Y}}(f(x))-f\left(G_{XY}(x)\right){\Vert }_1\right]. \end{gathered}$$ (4)

This geometry-consistency loss can be seen as a reconstruction loss that relies on the predefined geometric transformation function f(·). In this paper, we only take two common geometric transformations as examples, namely vertical flipping (vf) and 90° clockwise rotation (rot), to demonstrate the effectiveness of our geometry-consistency constraint. Note that, G_XY and $G_{\tilde{X}\tilde{Y}}$ have the same architecture and share all the parameters.

Full objective.

By combining our geometry-consistency constraint with the standard adversarial constraint, a remarkable one-sided unsupervised domain mapping can be targeted. The full objective for our GcGAN LGcGAN ${\mathcal{L}}_{GcGAN}\left(G_{XY},G_{\tilde{X}\tilde{Y}},D_Y,D_{\tilde{Y}},X,Y\right)$ will be:

$$\begin{gathered} {\mathcal{L}}_{GcGAN}={\mathcal{L}}_{gan}\left(G_{XY},D_Y,X,Y\right) \\ +{\mathcal{L}}_{gan}\left(G_{\tilde{X}\tilde{Y}},D_{\tilde{Y}},X,Y\right) \\ +\lambda {\mathcal{L}}_{geo}\left(G_{XY},G_{\tilde{X}\tilde{Y}},X,Y\right), \end{gathered}$$ (5)

where λ (λ = 20.0 in all the experiments) is a trade-off hyperparameter to weight the contribution of ${\mathcal{L}}_{gan}$ and ${\mathcal{L}}_{geo}$ during the model training. Carefully tuning λ may give preferable results to specific translation tasks.

Network architecture.

The full framework of our GcGAN is illustrated in Figure 2. Our experimental settings, network architectures, and learning strategies follow CycleGAN. We employ the same discriminator and generator as CycleGAN depending on the specific tasks. Specifically, the generator is a standard encoder-decoder, where the encoder contains two convolutional layers with stride 2 and several residual blocks [22] (6/9 blocks with respect to 128 × 128/256 × 256 of input resolution), and the decoder contains two deconvolutional layers also with stride 2. The discriminator distinguishes images at the patch level following PatchGANs [26, 33]. Like CycleGAN, we also use an identity mapping loss [55] in all of our experiments (except SVHN → MNIST), including our baseline (GAN alone). For other details, we use LeakyReLU as nonlinearity for the discriminators and instance normalization [57] to normalize convolutional feature maps.

Learning and inference.

We use the Adam solver [29] with a learning rate of 0.0002 and coefficients of (0.5, 0.999), where the latter is used to compute running averages of gradients and their squares. The learning rate is fixed in the initial 100 epochs, and linearly decays to zero over the next 100 epochs. Following CycleGAN, the negative log likelihood objective is replaced with a more stable and effective least-squares loss [43] for ${\mathcal{L}}_{gan}$. The discriminator is updated with random samples from a history of generated images stored in an image buffer [54] of size 50. The generator and discriminator are optimized alternately. In the inference phase, we feed an image only into the learned generator G_XY to obtain a translated image.

5. Experiments

We apply our GcGAN to a wide range of applications and make both quantitative and qualitative comparisons with the baseline (GAN alone) and previous state-of-theart methods including DistanceGAN and CycleGAN. We also study different ablations (based on rot) to analyze our geometry-consistency constraint. Since adversarial networks are not always stable, every independent experiment could result in slightly different scores. The scores in the quantitative analysis are computed by the average on three independent experiments.

5.1. Quantitative Analysis

The results demonstrate that our geometry-consistency constraint can not only partially filter out the candidate solutions having mode collapse or semantic distortions and thus produce more sensible translations, but also compatible with other unsupervised constraints such as cycle consistency [66] and distance preservation [5].

Cityscapes.

Cityscapes [12] contains 3975 image-label pairs, with 2975 used for training and 500 for validation (test in this paper). For a fair comparison with CycleGAN, the translators are trained at a resolution of 128 × 128 in an unaligned fashion. We evaluate our domain mappers using FCN scores and scene parsing metrics following previous works [41, 12, 66]. Specifically, for parsing → image, we assume that a high-quality translated image should produce qualitative semantic segmentation like real images when feeding it into a scene parser. Thus, we employ the pretrained FCN-8s [41] provided by pix2pix [26] to predict semantic labels for the 500 translated images. The label maps are then resized to the original resolution of 1024 × 2048 and compared against the ground truth labels using some standard scene parsing metrics including pixel accuracy, class accuracy, and mean IoU [41]. For image → parsing, since the fake labels are in the RGB format, we simply convert them into class-level labels using the nearest neighbor search strategy. In particular, we have 19 (category labels) + 1 (ignored label) categories for Cityscapes, each with a corresponding color value (RGB). For a pixel i in a translated parsing, we compute the distances between the 20 groundtruth color values and the color value of pixel i. The label of pixel i should be the one with the small est distance. Then, the aforementioned metrics are used to evaluate our mapping on the 19 category labels.

The parsing scores are presented in Table 1. Our Gc-GAN outperforms the baseline (GAN alone) by a large margin. We take the average of pixel accuracy, class accuracy, and mean IoU as the final score for analysis [65], i.e., score = (pixel acc + class acc + mean IoU)/3. For image → parsing, GcGAN (32.6%) yields a slightly higher score than CycleGAN (32.0%). For parsing → image, Gc-GAN (29.0% ~ 29.5%) obtains a convincing improvement of 1.3% ~ 1.8% over distanceGAN (27.7%).

Table 1: Parsing scores on Cityscapes.

${\mathcal{L}}_{GcGAN}$: The objective in Eqn. 5 with rot. GcGAN-rot-Separate: G_XY and $G_{\tilde{X}\tilde{Y}}$ do not share parameters. GcGAN-Mix-comb: Training GcGAN with both vf and rot in each iteration. GcGAN-Mix-rand: Training GcGAN with randomly chosen vf and rot in each iteration. GcGAN-rot + Cycle: GcGAN-rot with the cycle-consistency constraint.

method	image → parsing			parsing → image
	pixel acc	class acc	mean IoU	pixel acc	class acc	mean IoU
Benchmark Performance
CoGAN [40]	0.45	0.11	0.08	0.40	0.10	0.06
BiGAN/ALI [15, 16]	0.41	0.13	0.07	0.19	0.06	0.02
SimGAN [54]	0.47	0.11	0.07	0.20	0.10	0.04
CycleGAN (Cycle) [66]	0.58	0.22	0.16	0.52	0.17	0.11
DistanceGAN [5]	-	-	-	0.53	0.19	0.11
GAN alone (baseline)	0.514	0.160	0.104	0.437	0.161	0.098
GcGAN-rot	0.574	0.234	0.170	0.551	0.197	0.129
GcGAN-vf	0.576	0.232	0.171	0.548	0.196	0.127
Ablation Studies (Robustness & Compatibility)
${\mathcal{L}}_{GcGAN}\text{w/o\hspace{0.17em}}{\mathcal{L}}_{geo}(\lambda =0)$	0.486	0.163	0.102	0.396	0.148	0.088
${\mathcal{L}}_{GcGAN}\text{w/o\hspace{0.17em}}{\mathcal{L}}_{gan}(\tilde{X},\tilde{Y})$	0.549	0.199	0.139	0.526	0.184	0.111
GcGAN-rot-Seperate	0.575	0.233	0.170	0.545	0.196	0.124
GcGAN-Mix-comb	0.573	0.229	0.168	0.545	0.197	0.128
GcGAN-Mix-rand	0.564	0.217	0.156	0.547	0.192	0.123
GcGAN-rot + Cycle	0.587	0.246	0.182	0.557	0.201	0.132

We next perform ablation studies to further discuss Gc-GAN. The scores are reported in Table 1. Specifically, GcGAN-rot-Seperate shows that the generator G_XY employed in GcGAN is sufficient to handle both the style transfers (without shape deformation) X → Y and $\tilde{X}\to \tilde{Y}$. GcGAN-Mix-{comb, rand} demonstrate that persevering a geometric transformation can filter out most of the candidate solutions having mode collapse or undesired shape deformation, but preserving more ones can not leach more. Besides, GcGAN-Mix-rand performs slightly worse than GcGAN-Mix-comb. One of the possible reasons is that neither X_rot→Y_rot nor X_vf→Y_vf are sufficiently trained in the random case, which would decrease the effect of the aforementioned co-regularization mechanism. For GcGAN-rot + Cycle, we set the trade-off parameter for ${\mathcal{L}}_{cyc}$ to 10.0 as published in CycleGAN. The consistent improvement is a credible support that our geometry-consistency constraint is compatible with the widely-used cycle-consistency constraint. Moreover, when setting λ = 0 in LGcGAN, both G_XY and G_YX perform badly. One of the possible reasons is that, without the geometry consistency constraint, jointly modeling X→Y and $\tilde{X}\to \tilde{Y}$ with the shared generator G_XY would decrease the performance due to domain diversities caused by the geometric transformations. When removing ${\mathcal{L}}_{gan}\left(G_{\tilde{X}}{}_{\tilde{Y}},D_{\tilde{Y}}\right)$, the obtained scores are much higher than baseline (GAN alone) because $Y^{\prime }$ can partially correct ${\tilde{Y}}^{\prime }$ so that G_XY is able to handle the mapping $\tilde{X}\to \tilde{Y}$, and then ${\tilde{Y}}^{\prime }$ can constrain the mapping X→Y. As the analysis, when learning both ${\mathcal{L}}_{gan}\left(G_{XY},D_Y\right)$ and ${\mathcal{L}}_{gan}\left(G_{\tilde{X}\tilde{Y}},D_{\tilde{Y}}\right)$ with ${\mathcal{L}}_{geo}$, the co-regularization help gen erate preferable translations.

SVHN → MNIST.

We apply our approach to the SVHN → MNIST translation task. The translation models are trained on 73257 and 60000 training images of resolution 32 × 32 contained in the SVHN and MNIST training sets, respectively. The experimental settings follow DistanceGAN [5], including the default trade-off parameters for ${\mathcal{L}}_{cyc}$ and ${\mathcal{L}}_{dis}$. We compare our GcGAN with both DistanceGAN and CycleGAN in this translation task. To obtain quantitative results, we feed the translated images into a pretrained classifier trained on the MNIST training split, as done in [5].

Classification accuracies are reported in Table 2. Both GcGAN-rot and GcGAN-vf outperform DistanceGAN and CycleGAN by a large margin (about 6% ~ 7%). From the ablations, adding our geometry-consistency constraint to current unsupervised domain mapping frameworks will achieve different levels of improvements against the original ones. Note that, it seems that the distance-preservation constraint is not compatible with the cycle-consistency constraint on this task, but our geometry-consistency constraint can improve both ones.

Table 2:

Quantitative scores for SVHN → MNIST.

method	class acc (%)
Benchmark Performance
DistanceGAN (Dist.) [5]	26.8
CycleGAN (Cycle) [66]	26.1
Self-Distance [5]	25.2
GcGAN-rot	32.5
GcGAN-vf	33.3
Ablation Studies (Compatibility)
Cycle + Dist. [5]	18.0
GcGAN-rot + Dist.	34.0
GcGAN-rot + Cycle	33.8
GcGAN-rot + Dist. + Cycle	33.2

Google Maps.

We obtain 2194 (map, aerial photo) pairs of images in and around New York City from Google Maps [26], and split them into training and test sets with 1096 and 1098 pairs, respectively. We train Map ⇌ Aerial photo translators with an image size of 256×256 using the training set in an unsupervised manner (unpaired) by ignoring the pair information. For Aerial photo → Map, we make comparisons with CycleGAN using average RMSE and pixel accuracy (%). Given a pixel i with the ground-truth RGB value (r_i, g_i, b_i) and the predicted RGB value $\left(r_i^{\prime },g_i^{\prime },b_i^{\prime }\right)$, if $max\left(\left|r_i-r_i^{\prime }\right|,\left|g_i-g_i^{\prime }\right|,\left|b_i-b_i^{\prime }\right|\right)<\delta$, we consider this is an accurate prediction. Since maps only contain a limited number of different RGB values, it is reasonable to compute pixel accuracy using this strategy (δ₁ = 5 and δ₂ = 10 in this paper). For Map → Aerial photo, we only show some qualitative results in Figure 3.

Figure 3:

Qualitative comparison on Cityscapes (Parsing ⇌ Image) and Google Maps (Map ⇌ Aerial photo). GAN alone suffers from mode collapse. Translated images by GcGAN contain more details. GcGAN = GAN alone + geometry consistency.

From the scores presented in Table 3, GcGAN produces superior translations to the baseline (GAN alone). In particular, GcGAN yields an 18.0% ~ 21.9% improvement over the baseline with respect to pixel accuracy when δ = 5.0, demonstrating that the fake maps obtained by our GcGAN contain more details. In addition, GcGANs achieve competitive scores compared with CycleGAN.

Table 3:

Quantitative scores for Aerial photo → Map.

method	RMSE	acc (δ1)	acc (δ2)
Benchmark Performance
CycleGAN [66]	28.15	41.8	63.7
GAN alone (baseline)	33.27	19.3	42.0
GcGAN-rot	28.31	41.2	63.1
GcGAN-vf	28.50	37.3	58.9
Ablation Studies (Robustness & Compatibility)
GcGAN-rot-Separate	30.25	40.7	60.8
GcGAN-Mix-comb	27.98	42.8	64.6
GcGAN-rot + Cycle	28.21	40.6	63.5

5.2. Qualitative Evalutation

The qualitative results are shown in Figure 3, Figure 4, and Figure 5. Our geometry-consistency constraint improve the training of GAN alone, and helps to generate empirically more impressive translations on various applications. The following applications are trained in the image size of 256 × 256 with the rot geometric transformation.

Figure 4:

Qualitative comparison for SVHN → MNIST.

Figure 5:

Qualitative results on different applications, including Horse → Zebra, Monet → Photo, Synthetic ⇌ Real, Summar ⇌ Winter, Photo → Artist Painting, and Day ⇌ Night. GcGAN has the potential to produce realistic images. Zoom in for better view.

Horse → Zebra.

We apply GcGAN to the widely studied object transfiguration application task, i.e., Horse → Zebra. The images are randomly sampled from ImageNet [13] using the keywords (i.e., wild horse and zebra). The numbers of training images are 939 and 1177 for horse and zebra, respectively. We find that training GcGAN without parameter sharing would produce preferable translations for the task.

Synthetic ⇌ Real.

We employ the 2975 training images from Cityscapes as the real-world scenes, and randomly select 3060 images from SYNTHIA-CVPR16 [49], which is a virtual urban scene benchmark, as the synthetic images.

Summer ⇌ Winter.

The images used for the season translation tasks are provided by CycleGAN. The training set sizes for Summer and Winter are 1273 and 854.

Photo ⇌ Artistic Painting.

We translate natural images to artistic paintings with different art styles, including Monet, Cezanne, Van Gogh, and Ukiyo-e. We also perform Gc-GAN on the translation task of Monet’s paintings → photographs. We use the photos and paintings (Monet: 1074, Cezanne: 584, Van Gogh: 401, Ukiyo-e: 1433, and Photographs: 6853) collected by CycleGAN for training.

Day ⇌ Night.

We randomly extract 4500 training images for both Day and Night from the 91 webcam sequences captured by [30].

6. Conclusion

In this paper, we propose to enforce geometry consistency as a constraint for unsupervised domain mapping, which can be viewed as a predefined geometric transformation f(·) preserving the geometry of a scene. The geometry-consistency constraint makes the translation networks on the original images and transformed images co-regularize each other, which not only provides an effective remedy to the mode collapse problem suffered by standard GANs, but also reduces the semantic distortions in the translation. We evaluate our model, i.e., the geometry-consistent generative adversarial network (GcGAN), both qualitatively and quantitatively in various applications. The experimental results demonstrate that GcGAN achieves competitive and sometimes even better translations than the state-of-the-art methods including DistanceGAN and CycleGAN. Finally, our geometry-consistency constraint is compatible with other well-studied unsupervised constraints.