Generalization Bounds for Domain Adaptation

Abstract

In this paper, we provide a new framework to study the generalization bound of the learning process for domain adaptation. We consider two kinds of representative domain adaptation settings: one is domain adaptation with multiple sources and the other is domain adaptation combining source and target data. In particular, we use the integral probability metric to measure the difference between two domains. Then, we develop the specific Hoeffding-type deviation inequality and symmetrization inequality for either kind of domain adaptation to achieve the corresponding generalization bound based on the uniform entropy number. By using the resultant generalization bound, we analyze the asymptotic convergence and the rate of convergence of the learning process for domain adaptation. Meanwhile, we discuss the factors that affect the asymptotic behavior of the learning process. The numerical experiments support our results.

1 Introduction

In statistical learning theory, one of the major concerns is to obtain the generalization bound of a learning process, which measures the probability that a function, chosen from a function class by an algorithm, has a sufficiently small error (cf. [1,2]). Generalization bounds have been widely used to study the consistency of the learning process [3], the asymptotic convergence of empirical process [4] and the learnability of learning models [5]. Generally, there are three essential aspects to obtain generalization bounds of a specific learning process: complexity measures of function classes, deviation (or concentration) inequalities and symmetrization inequalities related to the learning process (cf. [3, 4, 6, 7]).

It is noteworthy that the aforementioned results of statistical learning theory are all built under the assumption that training and test data are drawn from the same distribution (or briefly called the assumption of same distribution). This assumption may not be valid in many practical applications such as speech recognition [8] and natural language processing [9] in which training and test data may have different distributions. Domain adaptation has recently been proposed to handle this situation and it is aimed to apply a learning model, trained by using the samples drawn from a certain domain (source domain), to the samples drawn from another domain (target domain) with a different distribution (cf. [10, 11, 12, 13]).

This paper is mainly concerned with two variants of domain adaptation. In the first variant, the learner receives training data from several source domains, known as domain adaptation with multiple sources (cf. [14, 15, 16, 17]). In the second variant, the learner minimizes a convex combination of empirical source and target risk, termed as domain adaptation combining source and target data (cf. [13, 18])¹.

1.1 Overview of Main Results

In this paper, we present a new framework to study generalization bounds of the learning processes for domain adaptation with multiple sources and domain adaptation combining source and target data, respectively. Based on the resultant bounds, we then study the asymptotic behavior of the learning process for the two kinds of domain adaptation, respectively. There are three major aspects in the framework: the quantity that measures the difference between two domains, the deviation inequalities and the symmetrization inequalities that are both designed for the situation of domain adaptation².

Generally, in order to obtain the generalization bounds of a learning process, it is necessary to obtain the corresponding deviation (or concentration) inequalities. For either kind of domain adaptation, we use a martingale method to develop the related Hoeffding-type deviation inequality. Moreover, in the situation of domain adaptation, since the source domain differs from the target domain, the desired symmetrization inequality for domain adaptation should incorporate some quantity to reflect the difference. We then obtain the related symmetrization inequality incorporating the integral probability metric that measures the difference between the distributions of the source and the target domains.

Next, we present the generalization bounds based on the uniform entropy number for both kinds of domain adaptation. Finally, based on the resultant bounds, we give a rigorous theoretic analysis to the asymptotic convergence and the rate of convergence of the learning processes for both types of domain adaptation. Meanwhile, we give a comparison with the related results under the assumption of same distribution. We also present numerical experiments to support our results.

1.2 Organization of the Paper

The rest of this paper is organized as follows. Section 2 introduces the problems studied in this paper. Section 3 introduces the integral probability metric that measures the difference between the distributions of two domains. We introduce the uniform entropy number for the situation of multiple sources in Section 4. In Section 5, we present the generalization bounds for domain adaptation with multiple sources, and then analyze the asymptotic behavior of the learning process for this type of domain adaptation. The last section concludes the paper. In the supplement (part A), we discuss the relationship between the integral probability metric D(S, T) and the other quantities proposed in the existing works including the -divergence and the discrepancy distance. Proofs of main results of this paper are provided in the supplement (part B). We study domain adaptation combining source and target data in the supplement (part C) and then give a comparison with the existing works on the theoretical analysis of domain adaptation in the supplement (part D).

2 Problem Setup

We denote := × ⊂ ℝ^I × ℝ^J (1 ≤ k ≤ K) and := × ⊂ ℝ^I × ℝ^J as the k-th source domain and the target domain, respectively. Set L = I + J. Let and stand for the distributions of the input spaces (1 ≤ k ≤ K) and , respectively. Denote $g_{\ast }^{(S_k)}:{\mathcal{X}}^{(S_k)}\to {\mathcal{Y}}^{(S_k)}$ and $g_{\ast }^{(T)}:{\mathcal{X}}^{(T)}\to {\mathcal{Y}}^{(T)}$ as the labeling functions of (1 ≤ k ≤ K) and , respectively. In the situation of domain adaptation with multiple sources, the distributions (1 ≤ k ≤ K) and differ from each other, or $g_{\ast }^{(S_k)}(1\le k\le K)$ and $g_{\ast }^{(T)}$ differ from each other, or both of the cases occur. There are sufficient amounts of i.i.d. samples ${\mathbf{Z}}_1^{N_k}={\{{\mathbf{z}}_n^{(k)}\}}_{n=1}^{N_k}$ drawn from each source domain (1 ≤ k ≤ K) but little or no labeled samples drawn from the target domain .

Given w = (w₁, ···, w_K) ∈ [0, 1]^K with ${\sum }_{k=1}^K{w}_k=1$, let g_w ∈ be the function that minimizes the empirical risk

$${\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g)=\sum _{k=1}^K{w}_k{\text{E}}_{N_k}^{(S_k)}(\ell \circ g)=\sum _{k=1}^K\frac{w_k}{N_k}\sum _{n=1}^{N_k}\ell (g({\mathbf{x}}_n^{(k)}),{\mathbf{y}}_n^{(k)})$$ (1)

over with respect to sample sets ${\{{\mathbf{Z}}_1^{N_k}\}}_{k=1}^K$, and it is expected that g_w will perform well on the target expected risk:

$${\text{E}}^{(T)}(\ell \circ g):=\int \ell (g({\mathbf{x}}^{(T)}),{\mathbf{y}}^{(T)})d\text{P}({\mathbf{z}}^{(T)}),g\in \mathcal{G},$$ (2)

i.e., g_w approximates the labeling $g_{\ast }^{(T)}$ as precisely as possible.

In the learning process of domain adaptation with multiple sources, we are mainly interested in the following two types of quantities:

• ${\text{E}}^{(T)}(\ell \circ g_{\mathbf{w}})-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g_{\mathbf{w}})$, which corresponds to the estimation of the expected risk in the target domain from a weighted combination of the empirical risks in the multiple sources ${\{{\mathcal{Z}}^{(S_k)}\}}_{k=1}^K$;

• E^(T) (ℓ ∘ g_w) − E^(T) (ℓ ∘ g̃_*), which corresponds to the performance of the algorithm for domain adaptation with multiple sources,

where g̃_* ∈ is the function that minimizes the expected risk E^(T) (ℓ ∘ g) over .

Recalling (1) and (2), since

$${\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ {\stackrel{\sim }{g}}_{\ast })-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g_{\mathbf{w}})\ge 0,$$

we have

$$\begin{array}{l}{\text{E}}^{(T)}(\ell \circ g_{\mathbf{w}})={\text{E}}^{(T)}(\ell \circ g_{\mathbf{w}})-{\text{E}}^{(T)}(\ell \circ {\stackrel{\sim }{g}}_{\ast })+{\text{E}}^{(T)}(\ell \circ {\stackrel{\sim }{g}}_{\ast })\\ \le {\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ {\stackrel{\sim }{g}}_{\ast })-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g_{\mathbf{w}})+{\text{E}}^{(T)}(\ell \circ g_{\mathbf{w}})-{\text{E}}^T(\ell \circ {\stackrel{\sim }{g}}_{\ast })+{\text{E}}^T(\ell \circ {\stackrel{\sim }{g}}_{\ast })\\ \le 2\underset{g\in \mathcal{G}}{sup}\mid {\text{E}}^{(T)}(\ell \circ g)-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g)\mid +{\text{E}}^{(T)}(\ell \circ {\stackrel{\sim }{g}}_{\ast }),\end{array}$$ (3)

and thus

$$0\le {\text{E}}^{(T)}(\ell \circ g_{\mathbf{w}})-{\text{E}}^{(T)}(\ell \circ {\stackrel{\sim }{g}}_{\ast })\le 2\underset{g\in \mathcal{G}}{sup}\mid {\text{E}}^{(T)}(\ell \circ g)-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g)\mid .$$

This shows that the asymptotic behaviors of the aforementioned two quantities, when the sample numbers N₁, ···, N_K go to infinity, can both be described by the supremum

$$\underset{g\in \mathcal{G}}{sup}\mid {\text{E}}^{(T)}(\ell \circ g)-{\text{E}}_{\mathbf{w}}^{(S)}(\ell \circ g)\mid ,$$ (4)

which is the so-called generalization bound of the learning process for domain adaptation with multiple sources.

For convenience, we define the loss function class

$$\mathcal{F}:=\{\mathbf{z}\mapsto \ell (g(\mathbf{x}),\mathbf{y}):g\in \mathcal{G}\},$$ (5)

and call as the function class in the rest of this paper. By (1) and (2), given sample sets ${\{{\mathbf{Z}}_1^{N_k}\}}_{k=1}^K$ drawn from the multiple sources ${\{{\mathcal{Z}}^{(S_k)}\}}_{k=1}^K$ respectively, we briefly denote for any f ∈ ,

$${\text{E}}^{(T)}f:=\int f({\mathbf{z}}^{(T)})d\text{P}({\mathbf{z}}^{(T)});{\text{E}}_{\mathbf{w}}^{(S)}f:=\sum _{k=1}^K\frac{w_k}{N_k}\sum _{n=1}^{N_k}f({\mathbf{z}}_n^{(k)}).$$ (6)

Thus, we can equivalently rewrite the generalization bound (4) for domain adaptation with multiple sources as

$$\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}^{(T)}f-{\text{E}}_{\mathbf{w}}^{(S)}f\mid .$$ (7)

3 Integral Probability Metric

As shown in some prior works (e.g. [13, 16, 17, 18, 19, 20]), one of major challenges in the theoretical analysis of domain adaptation is how to measure the distance between the source domain and the target domain . Recall that, if differs from , there are three possibilities: differs from , or $g_{\ast }^{(S)}$ differs from $g_{\ast }^{(T)}$, or both of them occur. Therefore, it is necessary to consider two kinds of distances: the distance between and and the distance between $g_{\ast }^{(S)}$ and $g_{\ast }^{(T)}$.

In [13, 18], the -divergence was introduced to derive the generalization bounds based on the VC dimension under the condition of “λ-close”. Mansour et al. [20] obtained the generalization bounds based on the Rademacher complexity by using the discrepancy distance. Both quantities are aimed to measure the difference between two distributions and . Moreover, Mansour et al. [17] used the Rényi divergence to measure the distance between two distributions. In this paper, we use the following quantity to measure the difference of the distributions between the source and the target domains:

Definition 3.1

Given two domains , ⊂ ℝ^L, let z^(S) and z^(T) be the random variables taking value from and , respectively. Let ⊂ ℝ be a function class. We define

$$D_{\mathcal{F}}(S,T):=\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}^{(S)}f-{\text{E}}^{(T)}f\mid ,$$ (8)

where the expectations E^(S) and E^(T) are taken with respect to the distributions and , respectively.

The quantity D(S, T) is termed as the integral probability metric that plays an important role in probability theory for measuring the difference between the two probability distributions (cf. [23, 24, 25, 26]). Recently, Sriperumbudur et al. [27] gave the further investigation and proposed the empirical methods to compute the integral probability metric in practice. As mentioned by Müller [page 432, 25], the quantity D(S, T) is a semimetric and it is a metric if and only if the function class separates the set of all signed measures with μ( ) = 0. Namely, according to Definition 3.1, given a non-trivial function class , the quantity D(S, T) is equal to zero if the domains and have the same distribution.

In the supplement (part A), we discuss the relationship between the quantity D(S, T) and other quantities proposed in the previous works, and then show that the quantity D(S, T) can be bounded by the summation of the discrepancy distance and another quantity, which measure the difference between the input-space distributions and and the difference between the labeling functions $g_{\ast }^{(S)}$ and $g_{\ast }^{(T)}$, respectively.

4 The Uniform Entropy Number

Generally, the generalization bound of a certain learning process is achieved by incorporating the complexity measure of the function class, e.g., the covering number, the VC dimension and the Rademacher complexity. The results of this paper are based on the uniform entropy number that is derived from the concept of the covering number and we refer to [22] for more details.

The covering number of a function class is defined as follows:

Definition 4.1

Let be a function class and d is a metric on . For any ξ > 0, the covering number of at radius ξ with respect to the metric d, denoted by ( , ξ, d) is the minimum size of a cover of radius ξ.

In some classical results of statistical learning theory, the covering number is applied by letting d be the distribution-dependent metric. For example, as shown in Theorem 2.3 of [22], one can set d as the norm ${\ell }_1({\mathbf{Z}}_1^N)$ and then derive the generalization bound of the i.i.d. learning process by incorporating the expectation of the covering number, i.e., $\text{E}\mathcal{N}(\mathcal{F},\xi ,{\ell }_1({\mathbf{Z}}_1^N))$. However, in the situation of domain adaptation, we only know the information of source domain, while the expectation $\text{E}\mathcal{N}(\mathcal{F},\xi ,{\ell }_1({\mathbf{Z}}_1^N))$ is dependent on both distributions of source and target domains because z = (x, y). Therefore, the covering number is no longer applicable to our framework to obtain the generalization bounds for domain adaptation. By contrast, uniform entropy number is distribution-free and thus we choose it as the complexity measure of function class to derive the generalization bounds for domain adaptation.

For clarity of presentation, we give a useful notation for the following discussion. For any 1 ≤ k ≤ K, given a sample set ${\mathbf{Z}}_1^{N_k}:={\{{\mathbf{z}}_n^{(k)}\}}_{n=1}^{N_k}$ drawn from , we denote ${{\mathbf{Z}}^{\prime }}_1^{N_k}:={\{{{\mathbf{z}}^{\prime }}_n^{(k)}\}}_{n=1}^{N_k}$ as the ghost-sample set drawn from such that the ghost sample ${{\mathbf{z}}^{\prime }}_n^{(k)}$ has the same distribution as ${\mathbf{z}}_n^{(k)}$ for any 1 ≤ n ≤ N_k and any 1 ≤ k ≤ K. Denoting ${\mathbf{Z}}_1^{2N_k}:=\{{\mathbf{Z}}_1^{N_k},{{\mathbf{Z}}^{\prime }}_1^{N_k}\}$. Moreover, given any f ∈ and any w = (w₁, ···, w_K) ∈ [0, 1]^K with ${\sum }_{k=1}^K{w}_k=1$, we introduce a variant of the ℓ₁ norm:

$${\Vert f\Vert }_{{\ell }_1^{\mathbf{w}}({\{{\mathbf{Z}}_1^{2N_k}\}}_{k1}^K)}:=\sum _{k=1}^K\frac{w_k}{N_k}\sum _{n=1}^{N_k}\left(\mid f({\mathbf{z}}_n^{(k)})\mid +\mid f({{\mathbf{z}}^{\prime }}_n^{(k)})\mid \right).$$ (9)

It is noteworthy that the variant ${\ell }_1^{\mathbf{w}}$ of the ℓ₁ norm is still a norm on the functional space, which can be easily verified by using the definition of norm, so we omit it here. In the situation of domain adaptation with multiple sources, setting the metric d as ${\ell }_1^{\mathbf{w}}({\{{\mathbf{Z}}_1^{2N_k}\}}_{k=1}^K)$, we then define the uniform entropy number of with respect to the metric ${\ell }_1^{\mathbf{w}}({\{{\mathbf{Z}}_1^{2N_k}\}}_{k=1}^K)$ as

$$ln{\mathcal{N}}_1^{\mathbf{w}}(\mathcal{F},\xi ,2\sum _{k=1}^K{N}_k):=\underset{{\{{\mathbf{Z}}_1^{2N_k}\}}_{k=1}^K}{sup}ln\mathcal{N}\left(\mathcal{F},\xi ,{\ell }_1^{\mathbf{w}}({\{{\mathbf{Z}}_1^{2N_k}\}}_{k=1}^K)\right).$$ (10)

5 Domain Adaptation with Multiple Sources

In this section, we present the generalization bound for domain adaptation with multiple sources. Based on the resultant bound, we then analyze the asymptotic convergence and the rate of convergence of the learning process for such kind of domain adaptation.

5.1 Generalization Bounds for Domain Adaptation with Multiple Sources

Based on the aforementioned uniform entropy number, the generalization bound for domain adaptation with multiple sources is presented in the following theorem:

Theorem 5.1

Assume that is a function class consisting of the bounded functions with the range [a, b]. Let w = (w₁, ···, w_K) ∈ [0, 1]^K with ${\sum }_{k=1}^K{w}_k=1$. Then, given an arbitrary $\xi >D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$, we have for any $({\prod }_{k=1}^K{N}_k)\ge \frac{8{(b-a)}^2}{{({\xi }^{\prime })}^2}$ and any ε > 0, with probability at least 1 − ε,

$$\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}_{\mathbf{w}}^{(S)}f-{\text{E}}^{(T)}f\mid \le D_{\mathcal{F}}^{(\mathbf{w})}(S,T)+{\left(\frac{\left(ln{\mathcal{N}}_1^{\mathbf{w}}(\mathcal{F},{\xi }^{\prime }/8,2{\sum }_{k=1}^K{N}_k)-ln(\epsilon /8)\right)}{\frac{\left({\prod }_{k=1}^K{N}_k\right)}{32{(b-a)}^2\left({\sum }_{k=1}^K{w}_k^2({\prod }_{i\ne k}{N}_i)\right)}}\right)}^{\frac{1}{2}},$$ (11)

where ${\xi }^{\prime }=\xi -D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$ and

$$D_{\mathcal{F}}^{(\mathbf{w})}(S,T):=\sum _{k=1}^K{w}_k{D}_{\mathcal{F}}(S_k,T).$$ (12)

In the above theorem, we show that the generalization bound ${sup}_{f\in \mathcal{F}}\mid {\text{E}}^{(T)}f-{\text{E}}_{\mathbf{w}}^{(S)}f\mid$ can be bounded by the right-hand side of (11). Compared to the classical result under the assumption of same distribution (cf. Theorem 2.3 and Definition 2.5 of [22]): with probability at least 1 − ε,

$$\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}_{N}f-\text{E}f\mid \le O\left({\left(\frac{ln{\mathcal{N}}_1(\mathcal{F},\xi ,N)-ln(\epsilon /8)}{N}\right)}^{\frac{1}{2}}\right)$$ (13)

with E_N f being the empirical risk with respect to the sample set ${\mathbf{Z}}_1^N$, there is a discrepancy quantity $D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$ that is determined by the two factors: the choice of w and the quantities D(S_k, T) (1 ≤ k ≤ K). The two results will coincide if any source domain and the target domain match, i.e., D(S_k, T) = 0 holds for any 1 ≤ k ≤ K.

In order to prove this result, we develop the related Hoeffding-type deviation inequality and the symmetrization inequality for domain adaptation with multiple sources, respectively. The detailed proof is provided in the supplement (part B). By using the resultant bound (11), we can analyze the asymptotic behavior of the learning process for domain adaptation with multiple sources.

5.2 Asymptotic Convergence

In statistical learning theory, it is well-known that the complexity of the function class is the main factor to the asymptotic convergence of the learning process under the assumption of same distribution (cf. [3, 4, 22]).

Theorem 5.1 can directly lead to the concerning theorem showing that the asymptotic convergence of the learning process for domain adaptation with multiple sources:

Theorem 5.2

Assume that is a function class consisting of bounded functions with the range [a, b]. Let w = (w₁, ···, w_K) ∈ [0, 1]^K with ${\sum }_{k=1}^K{w}_k=1$. If the following condition holds:

$$\underset{N_1,\cdots ,N_K\to +\infty }{lim}\frac{ln{\mathcal{N}}_1^{\mathbf{w}}(\mathcal{F},{\xi }^{\prime }/8,2{\sum }_{k=1}^K{N}_k)}{\frac{\left({\prod }_{k=1}^K{N}_k\right)}{32{(b-a)}^2\left({\sum }_{k=1}^K{w}_k^2({\prod }_{i\ne k}{N}_i)\right)}}<+\infty$$ (14)

with ${\xi }^{\prime }=\xi -D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$, then we have for any $\xi >D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$,

$$\underset{N_1,\cdots ,N_K\to +\infty }{lim}Pr\left\{\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}^{(T)}f-{\text{E}}_{\mathbf{w}}^{(S)}f\mid >\xi \right\}=0.$$ (15)

As shown in Theorem 5.2, if the choice of w ∈ [0, 1]^K and the uniform entropy number $ln{\mathcal{N}}_1^{\mathbf{w}}(\mathcal{F},{\xi }^{\prime }/8,2{\sum }_{k=1}^K{N}_k)$ satisfy the condition (14) with ${\sum }_{k=1}^K{w}_k=1$, the probability of the event that $“{sup}_{f\in \mathcal{F}}\mid {\text{E}}^{(T)}f-{\text{E}}_{\mathbf{w}}^{(s)}f\mid >\xi ”$ will converge to zero for any $\xi >D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$, when the sample numbers N₁, ···, N_K of multiple sources go to infinity, respectively. This is partially in accordance with the classical result of the asymptotic convergence of the learning process under the assumption of same distribution (cf. Theorem 2.3 and Definition 2.5 of [22]): the probability of the event that “sup_f∈ |Ef − E_N f| > ξ” will converge to zero for any ξ > 0, if the uniform entropy number ln ( , ξ, N) satisfies the following:

$$\underset{N\to +\infty }{lim}\frac{ln{\mathcal{N}}_1(\mathcal{F},\xi ,N)}{N}<+\infty .$$ (16)

Note that in the learning process of domain adaptation with multiple sources, the uniform convergence of the empirical risk on the source domains to the expected risk on the target domain may not hold, because the limit (15) does not hold for any ξ > 0 but for any $\xi >D_{\mathcal{F}}^{(\mathbf{w})}(S,T)$. By contrast, the limit (15) holds for all ξ > 0 in the learning process under the assumption of same distribution, if the condition (16) is satisfied.

By Cauchy-Schwarz inequality, setting $w_k=\frac{N_k}{{\sum }_{k=1}^K{N}_k}(1\le k\le K)$ minimizes the second term of the right-hand side of (11) and then we arrive at

$$\underset{f\in \mathcal{F}}{sup}\mid {\text{E}}_{\mathbf{w}}^{(S)}f-{\text{E}}^{(T)}f\mid \le \frac{{\sum }_{k=1}^K{N}_k{D}_{\mathcal{F}}(S_k,T)}{{\sum }_{k=1}^K{N}_k}+{\left(\frac{(ln{\mathcal{N}}_1^{\mathbf{w}}(\mathcal{F},{\xi }^{\prime }/8,2{\sum }_{k=1}^K{N}_k)-ln(\epsilon /8)}{\frac{\left({\sum }_{k=1}^K{N}_k\right)}{32{(b-a)}^2}}\right)}^{\frac{1}{2}},$$ (17)

which implies that setting $w_k=\frac{N_k}{{\sum }_{k=1}^K{N}_k}(1\le k\le K)$ can result in the fastest rate of convergence and our numerical experiments presented in the next section also support this point (cf. Fig. 1).

Figure 1.

Domain Adaptation with Multiple Sources

6 Numerical Experiments

We have performed the numerical experiments to verify the theoretic analysis of the asymptotic convergence of the learning process for domain adaptation with multiple sources. Without loss of generality, we only consider the case of K = 2, i.e., there are two source domains and one target domain. The experiment data are generated in the following way:

For the target domain = × ⊂ ℝ¹⁰⁰×ℝ, we consider as a Gaussian distribution N(0, 1) and draw ${\{{\mathbf{x}}_n^{(T)}\}}_{n=1}^{N_T}(N_T=4000)$ from randomly and independently. Let β ∈ ℝ¹⁰⁰ be a random vector of a Gaussian distribution N(1, 5), and let the random vector R ∈ ℝ¹⁰⁰ be a noise term with R ~ N(0, 0.5). For any 1 ≤ n ≤ N_T, we randomly draw β and R from N (1, 5) and N(0, 0.01) respectively, and then generate $y_n^{(T)}\in \mathcal{Y}$ as follows:

$$y_n^{(T)}=\langle {\mathbf{x}}_n^{(T)},\beta \rangle +R.$$ (18)

The derived ${\{({\mathbf{x}}_n^{(T)},y_n^{(T)})\}}_{n=1}^{N_T}(N_T=4000)$ are the samples of the target domain and will be used as the test data.

In the similar way, we derive the sample set ${\{({\mathbf{x}}_n^{(1)},y_n^{(1)})\}}_{n=1}^{N_1}(N_1=2000)$ of the source domain = × ⊂ ℝ¹⁰⁰ × ℝ: for any 1 ≤ n ≤ N₁,

$$y_n^{(1)}=\langle {\mathbf{x}}_n^{(1)},\beta \rangle +R,$$ (19)

where ${\mathbf{x}}_n^{(1)}~N(0.5,1)$, β ~ N(1, 5) and R ~ N(0, 0.5).

For the source domain = × ⊂ ℝ¹⁰⁰ × ℝ, the samples ${\{({\mathbf{x}}_n^{(2)},y_n^{(2)})\}}_{n=1}^{N_2}(N_2=2000)$ are generated in the following way: for any 1 ≤ n ≤ N₂,

$$y_n^{(2)}=\langle {\mathbf{x}}_n^{(2)},\beta \rangle +R,$$ (20)

where ${\mathbf{x}}_n^{(2)}~N(2,5)$, β ~ N(1, 5) and R ~ N(0, 0.5).

In this experiment, we use the method of Least Square Regression to minimize the empirical risk

$${\text{E}}_w^{(S)}(\ell \circ g)=\frac{w}{N_1}\sum _{n=1}^{N_1}\ell (g({\mathbf{x}}_n^{(1)}),y_n^{(1)})+\frac{(1-w)}{N_2}\sum _{n=1}^{N_2}\ell (g({\mathbf{x}}_n^{(2)}),y_n^{(2)})$$ (21)

for different combination coefficients w ∈ {0.1, 0.3, 0.5, 0.9} and then compute the discrepancy $\mid E_w^{(S)}f-E_{N_T}^{(T)}f\mid$ for each N₁ + N₂. The initial N₁ and N₂ both equal to 200. Each test is repeated 30 times and the final result is the average of the 30 results. After each test, we increment both N₁ and N₂ by 200 until N₁ = N₂ = 2000. The experiment results are shown in Fig. 1.

From Fig. 1, we can observe that for any choice of w, the curve of $\mid E_w^{(S)}f-E_{N_T}^{(T)}f\mid$ is decreasing when N₁ + N₂ increases, which is in accordance with the results presented in Theorems 5.1 & 5.2. Moreover, when w = 0.5, the discrepancy $\mid E_w^{(S)}f-E_{N_T}^{(T)}f\mid$ has the fastest rate of convergence, and the rate becomes slower as w is further away from 0.5. In this experiment, we set N₁ = N₂ that implies that N₂/(N₁ + N₂) = 0.5. Recalling (17), we have shown that w = N₂/(N₁ + N₂) will provide the fastest rate of convergence and this proposition is supported by the experiment results shown in Fig. 1.

7 Conclusion

In this paper, we present a new framework to study the generalization bounds of the learning process for domain adaptation. We use the integral probability metric to measure the difference between the distributions of two domains. Then, we use a martingale method to develop the specific deviation inequality and the symmetrization inequality incorporating the integral probability metric. Next, we utilize the resultant deviation inequality and symmetrization inequality to derive the generalization bound based on the uniform entropy number. By using the resultant generalization bound, we analyze the asymptotic convergence and the rate of convergence of the learning process for domain adaptation.

We point out that the asymptotic convergence of the learning process is determined by the complexity of the function class measured by the uniform entropy number. This is partially in accordance with the classical result of the asymptotic convergence of the learning process under the assumption of same distribution (cf. Theorem 2.3 and Definition 2.5 of [22]). Moreover, the rate of convergence of this learning process is equal to that of the learning process under the assumption of same distribution. The numerical experiments support our results. Finally, we give a comparison with the previous works [13, 14, 15, 16, 17, 18, 20] (cf. supplement, part D).

It is noteworthy that by Theorem 2.18 of [22], the generalization bound (11) can lead to the result based on the fat-shattering dimension. According to Theorem 2.6.4 of [4], the bound based on the VC dimension can also be obtained from the result (11). In our future work, we will attempt to find a new distance between two distributions and develop the generalization bounds based on other complexity measures, e.g., Rademacher complexities, and analyze other theoretical properties of domain adaptation.