Weakly Supervised Disentanglement by Pairwise Similarities

Abstract

Recently, researches related to unsupervised disentanglement learning with deep generative models have gained substantial popularity. However, without introducing supervision, there is no guarantee that the factors of interest can be successfully recovered (Locatello et al. 2018). Motivated by a real-world problem, we propose a setting where the user introduces weak supervision by providing similarities between instances based on a factor to be disentangled. The similarity is provided as either a binary (yes/no) or real-valued label describing whether a pair of instances are similar or not. We propose a new method for weakly supervised disentanglement of latent variables within the framework of Variational Autoencoder. Experimental results demonstrate that utilizing weak supervision improves the performance of the disentanglement method substantially.

Introduction

Disentanglement learning is a task of finding latent representations that separate the explanatory factors of variations in the data (Bengio, Courville, and Vincent 2013). In recent years, several methods (Higgins et al. 2017; Kim and Mnih 2018; Chen et al. 2018; Lopez et al. 2018) have been proposed to solve disentanglement learning under the Variational Autoencoder (VAE) framework. However, most of these existing methods are unsupervised. In this paper, we focus on improving the disentangling performance by utilizing weak supervisions in terms of pairwise similarities.

Locatello et al. (2018) showed that unsupervised disentanglement learning is fundamentally impossible if no inductive biases on models and datasets are provided. Existing unsupervised methods control the implicit inductive biases by choosing the hyperparameters. However, the factor of interest is not guaranteed to be successfully recovered by only tuning the hyper-parameters. Providing strong supervisions with discrete or real-valued labels have been previously suggested (Narayanaswamy et al. 2017; Kulkarni et al. 2015). However, such supervision can be expensive to acquire.

Our method is motivated by a real-world problem. In this problem, we want to understand how the Computer Tomography (CT) images are related to the severity of Chronic Obstructive Pulmonary Disease (COPD), which is a devastating disease related to cigarettes smoking. Since COPD manifests itself as airflow limitation, its severity can be measured via spirometry (meaning the measuring of breath). However, the disease severity is usually measured by combining two (Vestbo et al. 2013) or three (Quanjer et al. 2012) spirometric measures. It is not obvious how we can represent disease severity with one real value. Therefore, we represent disease severity using real-valued pairwise similarities between subjects, which are computed based on spirometric measures. The available CT images and the pairwise similarities motivate us to develop a disentanglement method that utilizes pairwise similarities when analyzing images.

In this paper, we assume that we are provided a measure of similarity between instances based on a specific factor of interest, in addition to the observations. The pairwise similarity can be binary (yes/no) or real-valued and may only be provided for a few pairs of instances. The goal is to learn disentangled representations such that a subset of the latent variables explain the factor of interest, but do not convey information about other factors of variations. We propose to achieve this goal by constructing a VAE model that generates both the samples and the pairwise similarities based on latent representations. We achieve disentanglement by letting the pairwise similarities depend on a subset of the latent variables but independent of the other latent variables, and penalizing the information capacity of the dependent latent variables. Our empirical evaluations on several benchmark datasets and the COPD dataset show that providing pairwise similarities improves the performance of the disentanglement method substantially.

Contributions

We make the following contributions in this paper: (1) We design a latent variable model that enables a user to provide similarities between instances in the desired latent space. (2) The similarity can be a binary or real-valued value provided for all or a subset of the pairs of instances. We formulate the model with a VAE framework and propose an efficient algorithm to train the model. (3) We conduct extensive experiments on benchmark datasets and a real-world dataset. Experimental results demonstrate that introducing weak supervision improves the disentanglement performance in different tasks.

Background

β-VAE

The β-VAE (Higgins et al. 2017) is base for many disentanglement methods. It introduces the inductive bias by increasing the weight of the KL divergence term in the evidence lower bound (ELBO) objective function, defined as

$$\begin{gathered} {\mathcal{L}}_{\beta -VAE}=\underset{\mathit{\theta },\varphi }{\max }{\mathbb{E}}_{{\mathbf{x}}_n\sim p_{\text{data}}}[{\mathbb{E}}_{q_{\varphi }\left({\mathbf{z}}_n|{\mathbf{x}}_n\right)}[\log p_{\mathit{\theta }}\left({\mathbf{x}}_n|{\mathbf{z}}_n\right)] \\ -\beta {\mathcal{D}}_{KL}\left(q_{\varphi }\left({\mathbf{z}}_n|{\mathbf{x}}_n\right)\Vert p(\mathbf{z})\right)], \end{gathered}$$ (1)

where $\mathbf{X}={\left\{{\mathbf{x}}_n\right\}}_{n=1}^N$ and $\mathbf{Z}={\left\{{\mathbf{z}}_n\right\}}_{n=1}^N$ denote the observed samples and the corresponding latent variables respectively, and N is the number of samples. We use p_θ(x_n|z_n) and q_ϕ(z_n|x_n) to represent the decoder and encoder networks that are parameterized by θ and ϕ, respectively. We let ${\mathcal{D}}_{KL}(\cdot \Vert \cdot )$ denote the KL divergence and p(z) denote prior distribution for z. In this paper, we let p(z) be an isotropic unit Gaussian distribution. In the equation, β ≥ 1 is a hyperparameter that controls the weight for the KL divergence term.

Method

We assume that we have access to the noisy observations of the similarities for pairs of instances. We use $\mathbf{Y}={\left\{y_{ij}\right\}}_{(i,j)\in \mathcal{J}}$ to represent the set of observed similarities, where $\mathcal{J}\subseteq \{(i,j)|i,j\in \{1,\dots ,N\}\}$. Note that not all pairwise similarity labels are necessarily observed. We allow y_ij to be either binary (y_ij ∈ {0, 1}) or real-valued between 0 and 1, where a larger value of y_ij indicates a stronger similarity between x_i and x_j.

In the following sections, we first explain the general framework of our model. We then discuss how the pairwise similarities can be incorporated into the model. Finally, we introduce a regularization term that encourages disentanglement.

The General Framework

We assume that both X and Y are noisy observations; hence, we use a probabilistic approach to model uncertainty. We adopt the VAE framework (Kingma and Welling 2013) such that x_n is reconstructed based on the latent variables z_n. We assume that the latent variable z is divided into two subspaces, i.e., z = [z^(u), z^(v)], where z^(u) (with d^(u) dimensions) accounts for the latent variables relevant to the factors of interest, while z^(v) (with d^(v) dimensions) accounts for the rest of information. Since y_ij represents pairwise similarity based on the factors of interest, it is only dependent on the coordinates of the latent variables of x_i and x_j in the z^(u) subspace; i.e., $p\left(y_{ij}|{\mathbf{z}}_i,{\mathbf{z}}_j\right)=p(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})$. Therefore, the joint distribution of the observed instances and similarities has the following factorization,

$$p_{\mathit{\theta }}(\mathbf{X},\mathbf{Y}|\mathbf{Z})={\prod }_{n=1}^N{p}_{\mathit{\theta }}\left({\mathbf{x}}_n|{\mathbf{z}}_n\right){\prod }_{(i,j)\in \mathcal{J}}p(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)}).$$ (2)

This model can be represented using a graphical model as shown in Figure 1. In this equation, p_θ(x_n|z_n) represents the reconstruction model of the VAE framework. We explain $p(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})$ in the next section.

Figure 1:

The decoder model p(X, Y|Z).

Modeling Pairwise Similarity

We view y_ij as the noisy observation of the similarity between i’th and j’th instances, which can be either a binary or a real- value measurement. We use the following function to model conditional of y_ij for both cases,

$$p\left(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)}\right)=\frac{1}{\mathcal{C}}{\left(g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)}^{y_{ij}}{\left(1-g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)}^{1-y_{ij}},$$ (3)

where $\mathcal{C}$ is the normalization constant and g(·, ·) is a function encoding the strength of the similarity given the relevant latent variables ${\mathbf{z}}_i^{(u)}$ and ${\mathbf{z}}_j^{(u)}$. In Equation (3), when y_ij is a binary variable, g can be viewed as probability that a user labels y_ij as 1. Hence, we choose g to return a value between 0 and 1 and $\mathcal{C}=1$. When y_ij is real-valued between 0 and 1, Equation (3) enables us to compute the normalization constant in a closed form:

$$\begin{gathered} \mathcal{C}={\int }_0^1{\left(g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)}^{y_{ij}}{\left(1-g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)}^{1-y_{ij}}d{y}_{ij} \\ =\frac{2g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})-1}{\log \left(g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)-\log \left(1-g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})\right)}. \end{gathered}$$ (4)

We adopt the following form for g:

$$g\left({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)}\right)=\sigma \left({\eta }_1\left({\eta }_2-{\Vert {\mathbf{z}}_i^{(u)}-{\mathbf{z}}_j^{(u)}\Vert }_2^2\right)\right),$$ (5)

where η₁ and η₂ are positive real hyperparameters controling the “steepness” and “threshold” of the similarity, respectively; and σ(·) is the sigmoid function. When η₁ → ∞, $g({\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})$ can be regarded as a hard thresholding function indicating whether or not ${\Vert {\mathbf{z}}_i^{(u)}-{\mathbf{z}}_j^{(u)}\Vert }_2^2$ is smaller than η₂. We replace the hard thresholding function with a sigmoid function σ(·) to make sure this function differentiable. The Figure 3 shows that when ${\Vert {\mathbf{z}}_i^{(u)}-{\mathbf{z}}_j^{(u)}\Vert }_2^2$ is small, it is more likely to have a large y_ij and vice versa.

Figure 3:

Plot for $p(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)})$ for real-valued y_ij. We fix the thresholding hyperparameter η₂ = 2. When $\Vert {\mathbf{z}}_i^{(u)}-{\mathbf{z}}_j^{(u)}{\Vert }_2^2$ is small, it is more likely to have a large y_ij and vice versa. The hyperparameter η₁ controls the “steepness” of the distribution.

Disentanglement via Regularization

Our goal of disentanglement is to encode all information about the factor of interest into z^(u) and to prevent it from containing irrelevant information. The general idea is to limit the capacity of z^(u); hence, its capacity can be used only for the relevant factors. Similar to the β-VAE, we use a regularized ELBO that increases the weight of the KL divergence between the approximate posterior (i.e., $q_{\varphi }({\mathbf{z}}_n^{(u)}|{\mathbf{x}}_n)$) and the prior (i.e., p(z^(u))), but we do not impose extra regularization for z^(v). The regularization term is defined as

$$\begin{gathered} \mathcal{R}=-{\mathbb{E}}_{{\mathbf{x}}_n\sim p_{\text{data}}}\left[\beta {\mathcal{D}}_{KL}\left(q_{\varphi }({\mathbf{z}}_n^{(u)}|{\mathbf{x}}_n)\Vert p({\mathbf{z}}^{(u)})\right)\right] \\ -{\mathbb{E}}_{{\mathbf{x}}_n\sim p_{\text{data}}}\left[{\mathcal{D}}_{KL}\left(q_{\varphi }({\mathbf{z}}_n^{(v)}|{\mathbf{x}}_n)\Vert p({\mathbf{z}}^{(v)})\right)\right], \end{gathered}$$ (6)

where β ≥ 1 is a real-valued hyperparameter that controls the weight of KL divergence.

Overall Model

The overall objective can be written as follows,

$$\begin{gathered} \mathcal{L}=\underset{\mathit{\theta },\varphi }{\max }{\mathbb{E}}_{{\mathbf{x}}_n\sim p_{\text{data}}}\left[{\mathbb{E}}_{q_{\varphi }\left({\mathbf{z}}_n|{\mathbf{x}}_n\right)}\left[\log p_{\mathit{\theta }}\left({\mathbf{x}}_n|{\mathbf{z}}_n\right)\right]\right] \\ +{\mathbb{E}}_{(i,j)\in \mathcal{J}}\left[{\mathbb{E}}_{q_{\varphi }\left({\mathbf{z}}_i,{\mathbf{z}}_j|{\mathbf{x}}_i,{\mathbf{x}}_j\right)}\left[\log p\left(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)}\right)\right]\right]+\mathcal{R}, \end{gathered}$$ (7)

where $p\left(y_{ij}|{\mathbf{z}}_i^{(u)},{\mathbf{z}}_j^{(u)}\right)$ is defined in Equation (3) and $\mathcal{R}$ is defined in Equation (6). We use the encoder q(·|·), to disentangle the factors at test time. Since we only have access to the weak labels Y at training time, the encoder can only take x_n as an argument. We use stochastic gradient descent (SGD) to optimize for θ and ϕ.

Related Work

There have been several unsupervised methods for learning disentangled representations with VAE, including β-VAE (Higgins et al. 2017), factor VAE (Kim and Mnih 2018), β-TCVAE (Chen et al. 2018) and HCV (Lopez et al. 2018). These methods achieve disentanglement by encouraging latent variables to be independent with each other. With these methods, the users can impact the disentanglement results only by tuning the hyperparameter. However, without explicit supervision, it is difficult to control the correspondence between a learned latent variable and a semantic meaning, and it is not guaranteed that the factor of interest can always be successfully disentangled (Locatello et al. 2018). In contrast, our proposed method utilizes the pairwise similarities as explicit supervision, which encourages the model to disentangle the factor of interest.

There have been attempts to improve disentanglement performance by introducing supervision. Narayanaswamy et al. (2017) and Kulkarni et al. (2015) propose semi-supervised VAE methods that learn disentangled representation, by making use of partially observed class labels or real-value targets. Bouchacourt, Tomioka, and Nowozin (2018) introduces supervision via grouping the samples. Our proposed method utilizes pairwise similarities.

Gaussian Process Prior VAE (GPPVAE) (Casale et al. 2018) assigns a Gaussian process prior to the latent variables. It makes use of the pairwise similarities between instances, by modeling the covariances between instances with a kernel function. GPPVAE does not focus on learning disentangled representation. Besides, GPPVAE requires the covariance matrix to be positive semi-definite, and the complete covariance matrix is observed without any missing values. In practice, a user might fail to provide labels satisfying these requirements. Our proposed method allows unobserved similarities and does not require the similarity matrix to be positive semi-definite.

Dual Swap Disentangling (DSD) (Feng et al. 2018) and Generative Multi-view Model (Chen, Denoyer, and Artières 2018) are VAE and GAN models that make use of binary similarity labels, respectively. They both assume that the latent variables z can be separated into subspaces z^(u) and z^(v), which is similar to our proposed model. However, both methods assume that similar instances share similar z^(u), but do not force dissimilar instances to be encoded differently in z^(u). As shown in our experiments, DSD is likely to converge into a trivial solution that all instances share similar z^(u), despite the similarity labels. In contrast, our proposed model is able to make use of both binary and real-valued similarities and it avoids this trivial solution by utilizing both similarity and dissimilarity labels.

Experiments

In this section, we evaluate our method quantitatively and qualitatively. We perform experiments for both binary and real-value similarity values. Our method is compared against a few competing methods qualitatively in terms of recovering semantic factors for rotating object or identifying the labels on benchmark datasets, where we evaluate our approach quantitatively on the recovery of the ground-truth factors. Then we apply our method to analyze the real-world COPD dataset. Finally, we study the robustness of our method for the choice of hyperparameters, the proportion of the observed pairwise similarity and the noisiness of the observed similarities.

In the following, we first introduce datasets used for our experiments, followed by the discussion of the various quantitative metrics used in this paper. We then report the results of the experiments.

Datasets and Competitive Methods

We evaluate our methods on five datasets: MNIST (LeCun and Cortes 2010), Fashion-MNIST (Xiao, Rasul, and Vollgraf 2017), Yale Faces (Georghiades, Belhumeur, and Kriegman 2001), 3D chairs (Aubry et al. 2014) and 3D cars (Krause et al. 2013). The details of these datasets are summarized in Table 1. For each dataset, we generate a subset of pairwise similarities based on one ground-truth factor of variations, as shown in the table. Unless specified otherwise, we let the number of observed pairwise labels be 0.01% of the number of all possible pairs. For the MNIST and fashion-MNIST datasets, we define $y_{ij}=\mathbb{1}\left(t_i=t_j\right)$ where t_i and t_j are the ground-truth labels for the sample i and j, and $\mathbb{1}$ is the indicator function. For Yale faces, 3D chairs and 3D cars, we use the Gaussian RBF kernel to define the similarities, i.e., $y_{ij}=exp(-\delta {\left(t_i,t_j\right)}^2/{\sigma }^2)$. Since the ground-truth factors in all three datasets involve azimuth angles, we use δ to denote the difference between two azimuth angles, e.g., δ(350°, 20°) =30°.

Table 1:

The Dataset

Name	Training instances	Held-out instances	Image size	The ground-truth factor
MNIST	60, 000	10, 000	28 × 28 × 1	discrete labels
Fashion-MNIST	60, 000	10, 000	28 × 28 × 1	discrete labels
Yale Faces	1, 903	513	64 × 64 × 1	azimuth lighting
3D chairs	69, 131	17, 237	64 × 64 × 3	azimuth rotations
3D cars	14, 016	3, 552	64 × 64 × 3	azimuth rotations

In addition to regular VAE (Kingma and Welling 2013), we compare our proposed method with three disentanglement approaches based on VAE, including β-VAE (Higgins et al. 2017), factor VAE (Kim and Mnih 2018), β-TCVAE (Chen et al. 2018). As a supervised disentanglement method, we compare our approach with Dual Swap Disentangling (DSD) (Feng et al. 2018). The DSD is designed to analyze binary similarities and cannot be applied to real-valued similarities. To make all methods comparable, we use the same encoder and decoder architectures for all the methods, which include four convolutional layers and one fully connected layer. To select the hyperparameters for our method, we use 5-fold cross validation on the training data. Since most of the competing methods are unsupervised, we choose the hyperparameters for them that achieves the best performance on the held-out data, which is advantageous for the competing methods resulting in an over-estimation of their performances. We define the metrics for the performance in the following section.

Quantitative Comparison

In this section, we perform two quantitative experiments. One is computing the Mutual Information Gap (MIG), which is a popular metric for evaluation of the disentanglement method, and the second experiment is a prediction task.

Mutual Information Gap (MIG)

We evaluate the disentanglement performance by computing the Mutual Information Gap (MIG) as introduced in (Chen et al. 2018). Let t represent the ground-truth factor and $\mathcal{I}(\cdot ,\cdot )$ represent the mutual information between two random variables (with 1 or more dimensions). In our model, since we assume z^(u) is relevant to t, we expect $\mathcal{I}\left({\mathbf{z}}^{(u)};t\right)$ to be large; while $\mathcal{I}({\mathbf{z}}_{.d}^{(v)};t)$ to be small for each dimension d ∈ {1... d^(v)}. Therefore, we can measure the disentanglement by computing the mutual information gap, defined as

$$\frac{1}{\mathcal{H}(t)}\left(\mathcal{I}\left({\mathbf{z}}^{(u)};t\right)-\underset{d\in \{1\dots d^{(v)}\}}{\max }\mathcal{I}({\mathbf{z}}_{\cdot d}^{(v)};t)\right),$$ (8)

where $\mathcal{H}(\cdot )$ represents the entropy of a random variable. The values of $\mathcal{I}(\cdot ,\cdot )$ and $\mathcal{H}(\cdot )$ can be empirically estimated as explained in (Chen et al. 2018). For each dataset, the dimensionality of z^(u), denoted by d^(u), is shown in the Table 2. Our method directly produces the z^(u) and z^(v) terms that can be plugged into Equation (8). Since the competing methods are unsupervised, the choice of the indices for z^(u) and z^(v) is not clear. For those methods, we first rank all latent variables based on the mutual information with respect to the ground-truth. Then, we pick the top d^(u) random variables to form z^(u) and the remaining latent variables are assigned to z^(v). The MIG values are estimated on the held-out data.

Table 2:

MIG metrics on the held-out data

Dataset	d (u)	Proposed	VAE	β-VAE	Factor-VAE	TCVAE	DSD
MNIST	2	0.68	0.01	0.03	0.33	0.04	0.01
Fashion-MNIST	2	0.52	0.11	0.28	0.36	0.19	0.01
Yale Faces	1	0.42	0.02	0.07	0.06	0.29	N/A
3D chairs	2	0.37	0.02	0.15	0.11	0.08	N/A
3D cars	2	0.41	0.02	0.22	0.15	0.16	N/A

DSD is designed for analyzing binary similarities, and cannot analyze real-valued similarities.

The values in Table 2 report the MIG for various methods. Our proposed method achieves substantially higher MIG values than other approaches. It outperforms the second-best methods by more than 40% in all five datasets. The results illustrate the importance of introducing supervision in disentanglement tasks. Although DSD is a supervised method that is formulated to incorporate binary pairwise similarities, it fails to disentangle the ground-truth factor. We speculate that the failure is due to convergence to a trivial solution, as mentioned in the Related Work Section.

Prediction Task

We use z^(u) as an input to a regression or classification method to predict the ground truth. We use the 5 Nearest Neighbour (5-NN) method for both classification and regression. Table 3 reports the outcome for different datasets, measured by Cohen’s kappa (κ) and R² with respect to the ground-truth. We measure Cohen’s kappa rather than classification accuracy because it corrects for the possibility of the agreement occurring by chance. For both measurements, a higher value indicates a better performance. We observe that our proposed method outperforms the competing methods in all tasks. This implies that instances with similar ground-truth factors are located near each other in the latent space z^(u).

Table 3:

Prediction Performance

	Dataset	Proposed	VAE	β-VAE	Factor-VAE	TCVAE	DSD
κ	MNIST	.969	.494	.326	.704	.260	.030
	Fashion-MNIST	.857	.389	.460	.613	.482	.003
R 2	Yale Faces	.968	.397	.760	.699	.692	N/A
	3D chairs	.912	.155	.357	.224	.196	N/A
	3D cars	.584	.391	.418	.177	.110	N/A

DSD is designed for analyzing binary similarities, and cannot analyze real-valued similarities.

Qualitative Comparison

In this subsection, we illustrate the disentanglement performance of the proposed method via qualitative comparison. We use the results on the MNIST and 3D-chairs datasets as examples (for more experimental results, see the supplementary materials¹).

MNIST

Figure 4(a) demonstrates z^(u) of the held-out instances from the MNIST dataset. Different colors represent different class labels. Figure 4(b) shows a similar concept for the competing method that achieves the highest MIG value in Table 2. We observe that the proposed model is able to learn z^(u) such that it explains the ground-truth factor (i.e., the digit class). All ten classes are well separated in the latent space with distinct centers, and instances from the same class are located close to each other. As shown in Figure 4(b), the factor-VAE is also able to learn a disentangled representation. However, regions of the instances of digit 4 and 9 are overlapping in the latent space.

Figure 4:

Held-out instances in MNIST dataset.

To illustrate the performance of the generative model, we plot some images generated by the proposed and the competing method in Figure 6(a) and 6(b), respectively. We first randomly sample an image from the held-out data and encode it into z = [z^(u), z^(v)]. Then, we keep z^(v) constant and manipulate z^(u). Using the new code, we generate new images that are displayed at their corresponding locations. In Figure 6(a), we find that the writing styles of ten digits are similar. This implies that z^(u) only contains the information about the ground-truth factor and not the other factors of variation. In contrast, we observe changes in writing styles in Figure 6(b). The figure shows that the reconstructed digits have different thicknesses, angles, widths.

Figure 6:

Manipulating z^(u) on MNIST dataset.

3D-chairs

We repeat the same plotting process for the 3D-chairs dataset. The results are shown in Figures 5 and 7. Since the ground truth (i.e., azimuth ) is a cyclic value, the ideal shape of the latent variable should look like a ring, which is approximately captured by our method in Figure 5(a). For some images, it is more challenging to determine which direction the chair faces (some chairs are almost centrosymmetric). These images are encoded into the regions close to the origin. Without proper supervision, β-VAE is not able to fully recover the complex underlying structure of the ground-truth factor, as shown in Figure 5(b).

Figure 5:

Held-out instances in 3D-chairs dataset.

Figure 7:

Manipulating z^(u) on 3D-chairs dataset.

We manipulate z^(u) and generate the images in Figure 7. As shown in 7(a), we observe the images of chairs facing various directions, located at the ring displayed in Figure 5(a). In Figure 7(b), we observe that β-VAE can reconstruct the chair images facing left and right, but other reconstructed images are blurry.

COPD dataset

A real-world application of the proposed model is to analyze the COPD dataset. The purpose of this application is to identify factors in the Computer Tomography (CT) images of the chest that are related to disease severity. We applied our method on a large-scale dataset (over 9K patients), where all patients have CT images as well as spirometric measurements. We use the spirometric measures to construct pairwise real-value similarities using Radial Basis Function.

In the COPD dataset, a ground-truth measure for disease severity is not available. Therefore, we use ${\mathbf{z}}^{(u)}\in ℝ$ learned by our model to predict several clinical measurements of disease severity from different aspects, via a 5-nearest neighbor regression and classification. The clinical measurements include (1) FEV₁pp measuring how quickly one can exhale, (2) Emphesyma% measuring the percentage of destructed lung tissue, (3) GasTrap% indicating amount of gas trapped in lung, and (4) GOLD score which is a six-categorical value indicating the severity of airflow limitation. In Table 4, we report R² for the first three measurements and Cohen’s kappa coefficient (κ) for the last measurement. The results suggest that our method is better than the unsupervised approach in disentangling the disease factor, as it outperforms them in predicting various measures of disease severity.

Table 4:

Prediction Performance in the COPD dataset

		Proposed	VAE	β-VAE	Factor-VAE	TCVAE
R 2	FEV1pp	.431	.002	.013	.010	.040
	Emphesyma%	.441	.027	.252	.191	.081
	GasTrap%	.522	.104	.279	.067	.110
κ	GOLD	.236	.023	.089	.061	.088

Choice of Hyperparameters

To illustrate how the hyperparameter β affects the performance of our proposed method, we first plot generated images with an improperly chosen β in Figure 8. In this figure, we find all ten digits. However, unlike the results shown in Figure 6(a), the writing styles (thicknesses, angles, widths, sizes, etc.) of the generated digits change significantly. This implies a failure of disentanglement, because z^(u) explains some factors of variations other than the one of interest (i.e., digit class).

Figure 8:

Generative results of the proposed result (β = 1) on MNIST. With improperly chosen β, z^(u) might explain other factors of variations (thicknesses, angles, etc.) other than the digit class.

To find a proper β for each dataset, we vary β and conduct 5-fold cross validation with the training instances. We plot the mean log-likelihood (logp_θ(X, Y|Z)) of five validations sets in Figure 9. We observe that a maximum log likelihood is achieved with choices of β between 2 to 10, but the optimal β differs across datasets. We choose β that maximizes the log-likelihood for each dataset.

Figure 9:

The mean value for the log-likelihood with different β in 5-fold cross validation. For each dataset, we choose β that maximizes the log-likelihood.

We illustrate how the hyperparameters η₁ and η₂ affect the disentanglement performance in Figure 10. In Figure 10(a), we fix η₂ = 2 and vary η₁; while in Figure 10(b), we fix η₁ = 1e3 and vary η₂. Because the log-likelihood is a function of η₁ and η₂, we report the MIG metrics for the held-out data, instead. We observe that when η₁ ≥ 1e3 and η₂ ≥ 1., these hyperparameters have limited effects on the MIG metrics. In all other experiments, we choose η₁ = 1e3 and η₂ = 2.

Figure 10:

The Plot of MIG metrics versus η₁ and η₂. When η₁ ≥ 1e3 and η₂ ≥ 1., these hyperparameters have limited effect on the MIG metrics for all datasets. In other experiments, we fix η₁ = 1e3 and η₂ = 2.

Number of Pairwise Labels

We investigate how the number of pairwise labels affects the performance of our proposed model. In Figure 11, we plot the MIG metrics for the held-out data versus the proportion of observed pairwise labels in training. We observe that in general, with more pairwise labels provided, the disentanglement performance improves. However, as the proportion approaches 1e – 4, the rate of improvement tapers. In all other experiments, we fix the proportion to be 1e – 4.

Figure 11:

The plot of MIG metrics versus the proportion of pairwise labels observed. In general, with more pairwise labels observed, the disentanglement performance improves. In the experiments, we choose the proportion to be 1e − 4(0.01%).

Noisy Similarity Labels

In all previous experiments, we do not introduce noise to the pairwise similarity labels. In this section, we introduce noise controlled by the noise level γ. For binary labels, we flip the labels with probability γ. For real-valued similarities, we let γ be the variance of the Gaussian noise, i.e., we add Gaussian noise $ϵ\sim \mathcal{N}(0,\gamma )$ and clip the results. We observe Figure 12 that the performance of our proposed method deteriorates as the noise level increases. Our proposed method is sensitive to noisy labels. By comparing the results to values in Table 2, we conclude that when γ ≤ 0.1, our proposed method gives better or comparable MIG metrics than the competing methods.

Figure 12:

The plot of MIG metrics versus noise level γ. The performance of our proposed method deteriorates as the noise level increases. Our proposed method is sensitive to noisy labels.

Conclusion

In this paper, we investigate the disentanglement learning problem, assuming a user introduces weak supervision by providing similarities between instances based on a factor to be disentangled. The similarity is provided as either a discrete (yes/no) or real-valued label between 0 and 1, where a larger value indicates a stronger similarity. We propose a new formulation for weakly supervised disentanglement of latent variables within the Variational Auto-Encoder (VAE) framework. Experimental results on both benchmark and real-world datasets demonstrate that utilizing weak supervision improves the performance of VAE in disentanglement learning tasks.

Figure 2:

The encoder model q(Z|X).