A Survey on Multi-View Clustering

Abstract

Clustering is a machine learning paradigm of dividing sample subjects into a number of groups such that subjects in the same groups are more similar to those in other groups. With advances in information acquisition technologies, samples can frequently be viewed from different angles or in different modalities, generating multi-view data. Multi-view clustering, that clusters subjects into subgroups using multi-view data, has attracted more and more attentions. Although MVC methods have been developed rapidly, there has not been enough survey to summarize and analyze the current progress. Therefore, we propose a novel taxonomy of the MVC approaches. Similar to other machine learning methods, we categorize them into generative and discriminative classes. In discriminative class, based on the way of view integration, we split it further into five groups: Common Eigenvector Matrix, Common Coefficient Matrix, Common Indicator Matrix, Direct Combination and Combination After Projection. Furthermore, we relate MVC to other topics: multi-view representation, ensemble clustering, multi-task clustering, multi-view supervised and semi-supervised learning. Several representative real-world applications are elaborated for practitioners. Some benchmark multi-view datasets are introduced and representative MVC algorithms from each group are empirically evaluated to analyze how they perform on benchmark datasets. To promote future development of MVC approaches, we point out several open problems that may require further investigation and thorough examination.

I. INTRODUCTION

CLUSTERING [1] is a paradigm to divide the subjects into a number of groups such that subjects in the same groups are more similar to other subjects in the same group and dissimilar to the subjects in other groups. It is a fundamental task in machine learning, pattern recognition and data mining fields and has widespread applications. Once subgroups are obtained by clustering methods, many subsequent analytic tasks can be conducted to achieve different ultimate goals. Traditional methods cluster subjects on the basis of only a single set of features or a single information window of the subjects. When multiple sets of features are available for each individual subject, how these views can be integrated to help identify essential grouping structure is a problem of our concern in this paper, which is often referred to as multi-view clustering. A good example to understand the importance of multi-view clustering, or multi-view learning is “the blind men and the elephant” story where each blind man (a single view of the subject) may not acquire the true picture of the subject [2], thus only collecting multi-view data can recover the whole picture of the subject.

Multi-view data are very common in real-world applications in the big data era. For instance, a web page can be described by the words appearing on the web page itself and the words underlying the links pointing to the web page from other pages in nature. In multimedia content understanding, multimedia segments can be simultaneously described by their video signals from visual camera and audio signals from voice recorders. The existence of such multi-view data raised the interest of multi-view learning [3], [4], [5], which has been extensively studied in the semi-supervised learning setting. For unsupervised learning, particularly, multi-view clustering, single view based clustering methods cannot make an effective use of the multi-view information in various problems. For instance, a multi-view clustering problem may require to identify clusters of subjects that differ in each of the data views. In this case, concatenating features from the different views into a single union followed by a single-view clustering method may not serve the purpose. It has no mechanism to guarantee that the resultant clusters differ in all of the views because the grouping may be biased towards a view (or views) that yields a dominantly large number of features in the feature union. Multi-view clustering has thus attracted more and more attentions in the past two decades, which makes it necessary and beneficial to summarize the state of the art and delineate open problems to guide future advancement.

At first, we give the definition of multi-view clustering (MVC). MVC is a machine learning paradigm to classify similar subjects into the same group and dissimilar subjects into different groups by combining the available multi-view feature information, and to search for the consistent clusterings across different views. Similar to the categorization of clustering algorithms in [1], we divide the existing MVC methods into two categories: generative (or model-based) approaches and discriminative (or similarity-based) approaches. Generative approaches try to learn the fundamental distribution of the data and use generative models to represent the data with each model representing one cluster. Discriminative approaches directly optimize an objective function that involves pairwise similarities to minimize the average similarity within clusters and to maximize the average similarity between clusters. In discriminative clustering family, there are mainly three strategies to combine multiple views: assuming that all views share a similar structure, direct combination of the views, and combination after projection of each view. Due to the different similar structures shared, we further split those MVC methods based on the first strategy into three groups: (1) common eigenvector matrix based (mainly multi-view spectral clustering), (2) common coefficient matrix based (mainly multi-view subspace clustering), (3) common indicator matrix based (mainly multi-view nonnegative matrix factorization clustering). The complete taxonomy is shown in Fig. 1.

Fig. 1:

The taxonomy of multi-view clustering methods.

Similarly motivated by the multi-view real applications as MVC, multi-view representation, multi-view supervised, and multi-view semi-supervised learning methods have an inherently close relation with MVC. Therefore, the similarities and differences of these different learning paradigms are also worth discussing. An obvious commonality between them is that they all learn with multi-view information. However, their learning targets are different. Multi-view representation methods aim to learn a joint compact representation for subjects from all of the views whereas MVC aims to perform sample partitioning, and MVC is learned without any label information. In contrast, multi-view supervised and semi-supervised learning methods have access to all or part of the label information. Some of the view combination strategies in these related paradigms can be borrowed and adapted by MVC. In addition, the relationships among MVC, ensemble clustering, multi-task clustering are also elaborated in this review.

MVC has been applied to many scientific domains such as computer vision, natural language processing, social multimedia, bioinformatics, and health informatics. Although MVC has permeated into many fields and made great success in practice, there are still some open problems that limit its further advancement. We point out several open problems and hope they can be helpful to promote the development of MVC. With this survey, we hope that readers can have a more comprehensive view of the MVC development and what is beyond the current progress.

There has been an earlier MVC survey [6]. We describe the differences between that one and ours which necessitate this survey. First, that work summarized the methods corresponding to a subset of the methods in our discriminative category, but the generative category of methods is a non-negligible direction. The generative methods assume that each cluster comes from a specific distribution in each view and combine them together to conduct MVC. Since most of them are based on the EM algorithm or convex mixture model, they have some inherent advantages over discriminative methods, such as being capable of dealing with missing values or obtaining global optimal solutions. Second, we discuss the relationship between MVC and several related topics, such as multi-view representation learning, ensemble clustering, multi-task clustering, and multi-view supervised and semi-supervised learning. This discussion helps researchers to position MVC in a scientific context and potentially gain deeper insights into all these topics. Third, we summarize representative applications of the various MVC methods for reference by interested users. Fourth, in Sections II and III we examine the pros and cons of each class of MVC methods and give the circumstances for which they are suitable, and we conduct a comprehensive comparison over the representative MVC algorithm in each group to further analyze and verify the advantages and disadvantages of each group of MVC algorithms. Last but not least, we draw attention to certain open problems with the hope that these directions help further advance MVC.

The remainder of this paper is organized as follows. In section II, we review the existing generative methods for MVC. Section III introduces several classes of discriminative MVC methods. In Section IV, we analyze the relationships between MVC and several related topics. Section V presents the applications of MVC in different areas. In Section VI, we introduce several commonly-used MVC datasets and conduct some experiments on them to investigate how they perform. In Section VII, we list several open problems with the aim to help advance the development of MVC. Finally, we make the conclusions.

II. GENERATIVE APPROACHES

Generative approaches aim to learn the generative models each of which is used to generate the data from a cluster. In multi-view case, multiple generative models need to be learned and then combined to obtain the final clustering results. In most cases, generative clustering approaches are based on mixture models or constructed via expectation maximization (EM) [7]. Therefore, we first introduce mixture models, EM algorithm and another popular single-view clustering model named convex mixture models (CMMs) [8], and then introduce the multi-view variants of these methods.

A. Mixture Models and CMMs

A generative approach assumes that data are sampled independently from a mixture model of multiple probability distributions. The mixture distribution can be written as

$$p(x\mid \theta )=\sum _{k=1}^K{\pi }_{k}p\left(x\mid {\theta }_k\right),$$ (1)

where ${\pi }_k$ is the prior probability of the kth component and satisfies ${\pi }_k\ge 0$, and ${\sum }_{k=1}^K{\pi }_k=1$, ${\theta }_k$ is the parameter of the kth probability density model and $\theta =\{\left({\pi }_k,{\theta }_k\right),k=1,2,\cdots ,K\}$ is the parameter set of the mixture model. For instance, ${\theta }_k=\left\{{\mu }_k,{\text{Σ}}_k\right\}$ for Gaussian mixture model.

EM is a widely used algorithm for parameter estimation of the mixture models. Suppose that the observed data and unobserved data are denoted by X and Z, respectively. {X, Z} and X are called complete data and incomplete data. In the E (expectation) step, the posterior distribution $p\left(Z\mid X,{\theta }^{old}\right)$ of the unobserved data is evaluated with the current parameter values ${\theta }^{old}$. The E step calculates the expectation of the complete data log likelihood evaluated for some general parameter value θ. The expectation, denoted by $Q\left(\theta ,{\theta }^{old}\right)$, is given by

$$Q\left(\theta ,{\theta }^{old}\right)=\sum _{Z}p\left(Z\mid X,{\theta }^{old}\right)\ln p(X,Z\mid \theta ).$$ (2)

The first item is the posterior distribution of the latent variables Z and the second one is the complete data log likelihood. According to maximum likelihood estimation, the M (maximization) step updates the parameters by maximizing the function (2)

$$\theta =\arg \underset{\theta }{\max }Q\left(\theta ,{\theta }^{old}\right).$$ (3)

Note that for clustering, X can be considered as the observed data while Z is the latent variable whose entry z_nk indicates the nth data point comes from the kth component. Also note that the posterior distribution form used to be evaluated in E step and the expectation of the complete data log likelihood used to evaluate the parameters are different for different distribution assumptions. It can adopt Gaussian distribution and any other probability distribution form, which depends on specific applications.

CMMs [8] are simplified mixture models that can probabilistically assign data points to clusters after extracting the representative exemplars from the data set. By maximizing the log-likelihood, all instances compete to become the “center” (representative exemplar) of the clusters. The instances corresponding to the components that received the highest priors are selected exemplars and then the remaining instances are assigned to the “closest” exemplar. The priors of the components are the only adjustable parameters of a CMM.

Given a data set $X=x_1,x_2,\cdots ,x_N\in {ℝ}^{d\times N}$, the CMM distribution is $Q(x)={\sum }_{j=1}^N{q}_j{f}_j(x)$, $x\in {ℝ}^d$, where $q_j\ge 0$ denotes the prior probability of the jth component that satisfies the constraint ${\sum }_{j=1}^N{q}_j=1$, and $f_j(x)$ is an exponential family distribution, with its expected parameters equal to the jth data point. Due to the bijection relationship between the exponential families and Bregman divergences [9], the exponential family $f_j(x)=C_{\varphi }(x)\exp \left(-\beta d_{\varphi }\left(x,x_j\right)\right)$ where $d_{\varphi }$ denotes the Bregman divergence that calculates the component distribution, $C_{\varphi }(x)$ is independent of x_j, and β is a constant controlling the sharpness of the components.

The log-likelihood that needs to be maximized is given as $L\left(X;{\left\{q_j\right\}}_{j=1}^N\right)=\frac{1}{N}{\sum }_{i=1}^N\log \left({\sum }_{j=1}^N{q}_j{f}_j\left(x_i\right)\right)=\frac{1}{N}{\sum }_{i=1}^N\log \left({{\sum }_{j=1}^N{q}_{j}e}^{-\beta d_{\varphi }\left(x_i,x_j\right)}\right)+\text{const}$. If the empirical samples are equally drawn, i.e., the prior of drawing each example is $\widehat{P}=1/N$, the log-likelihood can be equivalently expressed in terms of Kullback Leibler (KL) divergence between $\widehat{P}$ and Q(x) as

$$\begin{gathered} D(\widehat{P}\mid Q)=-\sum _{i=1}^N\widehat{P}\left(x_i\right)\log Q\left(x_i\right)-ℍ(\widehat{P}) \\ =-L\left(X;{\left\{q_j\right\}}_{j=1}^N\right)+\text{c} \end{gathered}$$ (4)

where $ℍ(\widehat{P})$ is the entropy of the empirical distribution $\widehat{P}(x)$ which does not depend on the parameter q_j, and c is a constant. Now, the problem is changed into minimizing (4), which is convex and can be solved by an iterative algorithm. In such an algorithm, the updating rule for prior probabilities is given by

$$q_j^{(t+1)}=q_j^{(t)}\sum _{i=1}^N\frac{\widehat{P}\left(x_i\right)f_j\left(x_i\right)}{{\sum }_{j^{\prime }=1}^N{q}_{j^{\prime }}^{(t)}{f}_{j^{\prime }}\left(x_i\right)}.$$ (5)

The data points are grouped into K disjoint clusters by requiring the instances with the K highest q_j values to serve as exemplars and then assigning each of the remaining instances to an exemplar with which the instance has the highest posterior probability. Note that the clustering performance is affected by the value of β. In [8] a reference value β₀ is determined using an empirical rule ${\beta }_0=N^2\log N/{\sum }_{i,j=1}^N{d}_{\varphi }\left(x_i,x_j\right)$ to identify a reasonable range of β, which is around β₀. More details refer to Paper [8].

B. MVC Based on Mixture Models or EM Algorithm

The method in [10] assumes that the two views are independent, and a multinomial distribution is adopted for document clustering problem. It uses the two-view case as an example, and executes the M and E steps on each view and then interchange the posteriors in two separate views in each iteration. The optimization process is terminated if the log-likelihood of observing the data does not reach a new maximum for a fixed number of iterations in each view. Based on different criteria and assumptions, two multi-view EM algorithm versions for finite mixture models are proposed in the paper [11].

Specifically, based on the CMMs for single-view clustering, the multi-view version proposed in [12] became much attractive because it can locate the global optimum and thus avoid the initialization and local optima problems of standard mixture models, which require multiple executions of the EM algorithms.

For multi-view CMMs, each x_i with m views is denoted by $\left\{x_i^1,x_i^2,\cdots ,x_i^m\right\}$, $x_i^v\in {ℝ}^{d^v}$, the mixture distribution for each view is given as $Q^v\left(x^v\right)={\sum }_{j=1}^N{q}_j{f}_j^v\left(x^v\right)=C_{\varphi }\left(x^v\right){\sum }_{j=1}^N{q}_{j}e-{\beta }^v{d}_{{\varphi }_v}\left(x^v,x_j^v\right)$. To pursue a common clustering across all views, all $Q^v\left(x^v\right)$ share the same priors. In addition, an empirical data set distribution ${\widehat{P}}^v\left(x^v\right)=1/N$, $x^v\in \left\{x_1^v,x_2^v,\cdots ,x_N^v\right\}$, is associated with each view and the multi-view algorithm minimizes the sum of KL divergences between ${\widehat{P}}^v\left(x^v\right)$ and $Q^v\left(x^v\right)$ across all views with the constraint ${\sum }_{j=1}^N{q}_j=1$. Thus, the formulated optimization problem is

$$\begin{array}{c}\underset{q_1,\cdots ,q_N}{\min }{\sum }_{v=1}^{m}D\left({\widehat{P}}^v\mid Q^v\right)=\underset{q_1,\cdots ,q_N}{\min }-{\sum }_{v=1}^m{\sum }_{i=1}^N{\widehat{P}}^v\left(x_i^v\right)\log Q^v\left(x_i^v\right)\\ -{\sum }_{v=1}^mℍ\left({\widehat{P}}^v\right).\end{array}$$ (6)

It is straightforward to see that the optimized objective is convex, hence the global minimum can be found. The prior update rule is given as follows:

$$q_j^{(t+1)}=\frac{q_j^{(t)}}{M}\sum _{v=1}^m\sum _{i=1}^N\frac{{\widehat{P}}^v{f}_j^v\left(x_i^v\right)}{{\sum }_{j^{\prime }=1}^N{q}_{j^{\prime }}^{(t)}{f}_{j^{\prime }}^v\left(x_i^v\right)}.$$ (7)

The prior q_j associated with the jth instance is a measure of how likely this instance is to be an exemplar, taking all views into account. The appropriate β^v values are identified in the range of an empirically defined ${\beta }_0^v$ by ${\beta }_0^v=N^2\log N/{\sum }_{i,j=1}^N{d}_{{\varphi }_v}\left(x_i^v,x_j^v\right)$. From Eq. (6), it can be found that all views contribute equally to the sum, without considering their different importance. To overcome this limitation, a weighted version of multi-view CMMs was proposed in [13].

1). Summary:

For the aforementioned MVC generative methods, we can find that linear combination with different weights to different views is a common way to fuse information. In addition, multi-view generative clustering has not attracted enough attention, maybe because the technique is more difficult compared with its discriminative counterpart. It is not easy for generative methods to combine views by sharing a common variable or distribution, but sharing common variable(s) is the most popular way to combine views in the discriminative paradigm. This can limit the development of multi-view generative clustering to some extent, but researchers are actively seeking for ways to combine views in multi-view generative clustering methods. For example, it is quite reasonable to share some commonality across the distributions of the data views corresponding to the same cluster. Moreover, generative methods have their advantages. Firstly, generative methods are based on data distribution, and if the data do follow the distribution assumed, the method should perform well. Secondly, given the methods, such as in [12] can get the global optimum, it is quite intriguing. Thirdly, there is no need to pre-specify the number of clusters. We believe multi-view generative clustering even single-view generative clustering method is an underestimated direction, more efforts can be made along this direction in the future.

III. DISCRIMINATIVE APPROACHES

Compared with generative approaches, discriminative approaches directly optimize the objective to seek for the best clustering solution rather than first modelling the sample distribution then solving these models to determine clustering result. Directly focusing on the objective of clustering makes discriminative approaches gain more attentions and develop more comprehensively. Up to now, most of existing MVC methods are discriminative approaches. Based on how to combine multiple views, we categorize MVC methods into five main classes and introduce the representative works in each group.

Given the data with N data points and m views, each data point x_i is denoted by $\left\{x_i^1,x_i^2,\cdots ,x_i^m\right\}$, $x_i^v\in {ℝ}^{d^v}$. The aim of MVC is to cluster the N data points into K classes. That is, finally we will get a membership matrix $H\in {ℝ}^{N\times K}$ to indicate which data points are in the same group while others in other classes, the sum of each row entries of H should be 1 to make sure each row is a probability distribution. If only one entry of each row is 1 and all others are 0, it is the so-called hard clustering otherwise it is soft clustering. In the following 5 subsections, we will introduce each class of multi-view discriminative clustering methods.

A. Common Eigenvector Matrix (Mainly Multi-View Spectral Clustering)

This class of MVC methods are based on a commonly used clustering technique spectral clustering. Since spectral clustering hinges crucially on the construction of the graph Laplacian [14], [15] and the resulting eigenvectors reflect the grouping structure of the data, this class of MVC methods guarantee to get a common clustering result by assuming that all the views share the same or similar eigenvector matrix. There are two representative methods: co-training spectral clustering [16] and co-regularized spectral clustering [17]. Before discussing them, we will introduce spectral clustering [18] first.

1). Spectral Clustering:

Spectral clustering is a clustering technique that utilizes the properties of graph Laplacian where the graph edges denote the similarities between data points and solve a relaxation of the normalized min-cut problem on the graph [19]. Compared with other widely used methods such as the k-means algorithm that only fits the spherical shaped clusters, spectral clustering can apply to arbitrary shaped clusters and demonstrate good performance.

Given $G=(V,E)$ as a weighted undirected graph with vertex set $V=v_1,\cdots ,v_N$. The data adjacency matrix of the graph is defined as W whose entry $w_{ij}$ represents the similarity of two vertices v_i and v_j. If $w_{ij}=0$, it means that the vertices v_i and v_j are not connected. Apparently W is symmetric since G is an undirected graph. The degree matrix D is defined as the diagonal matrix with the degrees $d_1,\cdots ,d_N$ of each vertex on the diagonal, where $d_i={\sum }_{j=1}^N{w}_{ij}$. Generally, the graph Laplacian is D – W and the normalized graph Laplacian is $\tilde{L}=D^{-1/2}(D-W)D^{-1/2}$. In many spectral clustering works e.g. [18], [16], [17], [20], $L=D^{-1/2}W{D}^{-1/2}$ is also used to change a minimization problem (9) into a maximization problem (8) since $L=I-\tilde{L}$ where I is the identity matrix. Following the same terminology adopted in [18], [16], [17], [20], we will name both L and $\tilde{L}$ as normalized graph Laplacians afterwards. Now the single-view spectral clustering approach can be formulated as follows:

$$\{\begin{array}{c}\underset{U\in {ℝ}^{N\times K}}{\max }tr\left(U^{\text{T}}LU\right)\\ s.t.U^{\text{T}}U=I,\end{array}$$ (8)

which is also equivalent to the following problem:

$$\{\begin{array}{c}\underset{U\in {ℝ}^{N\times K}}{\min }tr\left(U^{\text{T}}\tilde{L}U\right)\\ s.t.U^{\text{T}}U=I,\end{array}$$ (9)

where tr denotes the trace norm of a matrix. The rows of matrix U are the embeddings of the data points, which can be fed into the k-means to obtain the final clustering results. A version of the Rayleigh-Ritz theorem in [21] shows that the solution of the above optimization problem is given by choosing U as the matrix containing, respectively, the largest or smallest K eigenvectors of L or $\tilde{L}$ as columns. To understand the spectral clustering method better, we outline a commonly used algorithm [18] as follows:

• Construct the adjacency matrix W.

• Compute the normalized Laplacian matrix $L=D^{-1/2}W{D}^{-1/2}$.

• Calculate the eigenvectors of L and stack the top K eigenvectors as the columns to construct a N × K matrix U.

• Normalize each row of U to obtain $U_{sym}$.

• Run the k-means algorithm to cluster the row vectors of $U_{sym}$.

• Assign subject i to cluster k if the ith row of $U_{sym}$ is assigned to cluster k by the k-means algorithm.

Apart from the symmetric normalization operator $U_{sym}$, another normalization operator $U_{lr}=D^{-1}W$ is also commonly used. Refer to [22] for more details about spectral clustering.

2). Co-Training Multi-View Spectral Clustering:

For semi-supervised learning, co-training with two views has been a widely recognized idea when both labeled and unlabeled data are available. It assumes that the predictive models constructed in each of the two views will lead to the same labels for the same sample with high probability. There are two main assumptions to guarantee the success of co-training: (1) Sufficiency: each view is sufficient for sample classification on its own, (2) Conditional independence: the views are conditionally independent given the class labels. In the original co-training algorithm [23], two initial predictive functions f₁ and f₂ are trained in each view using the labeled data, then the following steps are repeatedly performed: the most confident examples predicted by f₁ are added to the labeled set to train f₂ and vice versa, then f₁ and f₂ are re-trained on the enlarged labeled datasets. It can be shown that after a number of iterations, f₁ and f₂ will agree with each other on labels.

For co-training multi-view spectral clustering, the motivation is similar: the clustering result in all views should agree. In spectral clustering, the eigenvectors of the graph Laplacian encode the discriminative information of the clustering. Therefore, co-training multi-view spectral clustering [16] uses the eigenvectors of the graph Laplacian in one view to cluster samples and then use the clustering result to modify the graph Laplacian in the other view.

Each column of the similarity matrix (also called the adjacency matrix) $W_{N\times N}$ can be considered as a N-dimensional vector that indicates the similarities of ith point with all the points in the graph. Since the largest K eigenvectors have the discriminative information for clustering, the similarity vectors can be projected along those directions to retain the discriminative information for clustering and throw away the within cluster details that might confuse the clustering. After that, the projected information is projected back to the original N-dimensional space to get the modified graph. Finally, k-means algorithm is conducted on most informative eigenvector matrix to get the final clustering result.

To make the co-training spectral clustering algorithm clear, we borrow Algorithm 1 from [16]. Note that the symmetrization operator sym on a matrix S is defined as sym $(S)=\left(S+S^{\text{T}}\right)/2$ in Algorithm 1.

Algorithm 1.

Co-training Multi-View Spectral Clustering

Input: Similarity matrices for two views: W(1) and W(2).

Output: Assignments to K clusters.

Initialize: ${\mathbf{L}}^{(v)}={\mathbf{D}}^{{(v)}^{(-1/2)}}{\mathbf{L}}^{(v)}{\mathbf{D}}^{{(v)}^{(-1/2)}}$ for v = 1,2, $U^{{(v)}^0}=\underset{U\in {ℝ}^{N\times K}}{\text{argmax}}tr\left(U^{\text{T}}{L}^{(v)}U\right)s.t.U^{\text{T}}U=I$ for v = 1,2.

for i=1 to t do

1. $S^{(1)}=sym\left(U^{{(2)}^{i-1}}{U}^{{(2)}^{{i-1}^{\text{T}}}}{W}^{(1)}\right)$

2. $S^{(2)}=sym\left(U^{{(1)}^{i-1}}{U}^{{(2)}^{{i-1}^{\text{T}}}}{W}^{(2)}\right)$

3. Use S(1) and S(2) as the new graph similarities and compute the graph Laplacians. Solve for the largest K eigenvectors to obtain $U^{{(1)}^i}$ and $U^{{(2)}^i}$

end for

4: Normalize each row of $U^{{(1)}^i}$ and $U^{{(2)}^i}$.

5: Form matrix $V=U^{{(v)}^i}$, where v is the most informative view a priori. If there is no prior knowledge on the view informativeness, matrix V can also be set to be columnwise concatenation of the two $U^{{(v)}^i}$s.

6: Assign example j to cluster K if the jth row of V is assigned to cluster K by the k-means algorithm.

3). Co-Regularized Multi-View Spectral Clustering:

Co-regularization is an effective technique in semi-supervised multi-view learning. The core idea of co-regularization is to minimize the distinction between the predictor functions of two views acting as one part of the objective function. However, there are no predictor functions in unsupervised learning like clustering, so how to implement the co-regularization idea in clustering problem? Co-regularized multi-view spectral clustering [17] adopted the eigenvectors of graph Laplacian to play the similar role of predictor functions in semi-supervised learning scenario and proposed two co-regularized clustering approaches.

Let $U^{(s)}$ and $U^{(t)}$ be the eigenvector matrices corresponding to any pair of view graph Laplacians $L^{(s)}$ and $L^{(t)}$ $(1\le s,t\le m,s\ne t)$. The first version uses a pairwise co-regularization criteria that enforces $U^{(s)}$ and $U^{(t)}$ as close as possible. The measure of clustering disagreement between the two views s and t is $D\left(U^{(s)},U^{(t)}\right)=\parallel \frac{K^{(s)}}{{\Vert K^{(s)}\Vert }_F^2}-\frac{K^{(t)}}{{\Vert K^{(t)}\Vert }_F^2}{\parallel }_F^2$, where $K^{(s)}=U^{(s)}{U}^{{(s)}^{\text{T}}}$ using linear kernel is the similarity matrix of $U^{(s)}$. Since ${\Vert K^{(s)}\Vert }_F^2=K$, where K is the number of the clusters, disagreement between the clustering solutions in the two views can be measured by $D\left(U^{(s)},U^{(t)}\right)=-tr\left(U^{(s)}{U}^{{(s)}^{\text{T}}}{U}^{(t)}{U}^{{(t)}^{\text{T}}}\right)$. Integrating the measure of the disagreement between any pair of views into the spectral clustering objective function, the pairwise co-regularized multi-view spectral clustering can be formed as the following optimization problem:

$$\{\begin{array}{c}\underset{U^{(1)},U^{(2)},\cdots ,U^{(m)}\in {ℝ}^{N\times K}}{\max }{\sum }_{s=1}^{m}tr\left(U^{{(s)}^{\text{T}}}{L}^{(s)}{U}^{(s)}\right)\\ +{\sum }_{1\le s,t\le m,s\ne t}\lambda tr\left(U^{(s)}{U}^{{(s)}^{\text{T}}}{U}^{(t)}{U}^{{(t)}^{\text{T}}}\right)\\ s.t.U^{(s)}{}^{{}^{\text{T}}}{U}^{(s)}=I,\forall 1\le s\le m.\end{array}$$ (10)

The hyperparameter λ is used to trade-off the spectral clustering objectives and the spectral embedding disagreement terms. After the embeddings are obtained, each U^s can be fed for k-means clustering method, the final results are marginally different.

The second version named centroid-based co-regularization enforces the eigenvector matrix from each view to be similar by regularizing them towards a common consensus eigenvector matrix. The corresponding optimization problem is formulated as

$$\{\begin{array}{c}\underset{U^{(1)},U^{(2)},\cdots ,U^{(m)},U^{\ast }\in {ℝ}^{N\times K}}{\max }{\sum }_{s=1}^{m}tr\left(U^{{(s)}^{\text{T}}}{L}^{(s)}{U}^{(s)}\right)\\ +{\lambda }_s{\sum }_{s=1}^{m}tr\left(U^{(s)}{U}^{(s)}{}^{{}^{\text{T}}}{U}^{(\ast )}{U}^{(\ast )}{}^{{}^{\text{T}}}\right)\\ s.t.U^{(s)}{}^{{}^{\text{T}}}{U}^{(s)}=I,\forall 1\le s\le m,U^{{\ast }^{\text{T}}}{U}^{\ast }=I.\end{array}$$ (11)

Compared with pairwise co-regularized version, centroid-based MVC does not need to combine the obtained eigenvector matrices of all views to run k-means. However, the centroid-based version possesses one potential drawback: the noisy views could potentially affect the optimal eigenvectors as it depends on all the views.

Cai et. al. [24] used a common indicator matrix across the views to perform multi-view spectral clustering and derived a formulation similar to the centroid-based co-regularization method. The main difference is that [24] used $tr((U^{(\ast )}-{U^{(s)})}^{\text{T}}\left(U^{(\ast )}-U^{(s)}\right))$ as the disagreement measure between each view eigenvector matrix and the common eigenvector matrix while co-regularized multi-view spectral clustering [17] adopted $tr\left(U^{(s)}{U}^{{(s)}^{\text{T}}}{U}^{(\ast )}{U}^{{(\ast )}^T}\right)$. The optimization problem [24] is formulated as

$$\{\begin{array}{c}\underset{U^{(s)},s=1,2\cdots ,m,U^{\ast }}{\max }{\sum }_{s=1}^{m}tr\left(U^{{(s)}^T}{L}^{(s)}{U}^{(s)}\right)\\ +\lambda {\sum }_1^{m}tr\left({\left(U^{\ast }-U^{(s)}\right)}^{\text{T}}\left(U^{\ast }-U^{(s)}\right)\right)\\ s.t.U^{\ast }\ge 0,U^{\ast \text{T}}{U}^{\ast }=I.\end{array}$$ (12)

where $U^{\ast }\ge 0$ makes $U^{\ast }$ become the final cluster indicator matrix. Different from general spectral clustering that get eigenvector matrix first and then run clustering (such as k-means that is sensitive to initialization condition) to assign clusters, Cai et al. [24] directly solves the final cluster indicator matrix, thus it will be more robust to the initial condition.

4). Others:

Besides the two representative multi-view spectral clustering methods discussed above, Wang et al. [25] enforces a common eigenvector matrix across the views and formulates a multi-objective problem which is then solved using Pareto optimization.

A relaxed kernel k-means can be shown to be equivalent to spectral clustering, see the following Subsection III-D2, Ye et al. [26] proposes a co-regularized kernel k-means for MVC. With a multi-layer Grassmann manifold interpretation, Dong et al. [27] obtains the same formulation with the pairwise co-regularized multi-view spectral clustering.

Because the MVC methods based on a shared eigenvector matrix are rooted from the special clustering, they can be applied to data clusters of any shape or any positioning of cluster centers. This merit is inherited from spectral clustering that does not make any assumption about the statistics of the clusters. However, since spectral clustering needs eigendecomposition, this type of MVC methods can be time consuming.

B. Common Coefficient Matrix (Mainly Multi-View Subspace Clustering)

In many practical applications, even though the given data are high dimensional, the intrinsic dimension of the problem is often low. For example, the number of pixels in a given image can be large, yet only a few parameters are used to describe the appearance, geometry, and dynamics of a scene. This motivates the development of finding the underlying low dimensional subspace. In practice, the data could be sampled from multiple subspaces. Subspace clustering [28] is the technique to find the underlying subspaces and then cluster the data points correctly according to the identified subspaces.

1). Subspace clustering:

Subspace clustering uses the self-expressiveness property [29] of the data samples, i.e., each sample can be represented by a linear combination of a few other data samples. The classic subspace clustering formulation is given as follows:

$$X=XZ+E$$ (13)

where $Z=\left\{z_1,z_2,\cdots ,z_N\right\}\in {ℝ}^{N\times N}$ is the subspace coefficient matrix (representation matrix), and each z_i is the representation of the original data point x_i based on the subspace. $E\in {ℝ}^{N\times N}$ is the noise matrix.

The subspace clustering can be formulated as the following optimization problem:

$$\{\begin{array}{c}\underset{Z}{\min }\parallel \mathit{X}-\mathit{X}\mathit{Z}{\parallel }_F^2\\ s.t.Z(i,i)=0,Z^{\text{T}}1=1.\end{array}$$ (14)

The constraint $Z(i,i)=0$ is used to avoid the case that a data point is represented by itself, while $Z^{\text{T}}1=1$ denotes that the data point lies in a union of affine subspaces. The nonzero elements of z_i correspond to data points from the same subspace.

After getting the subspace representation Z, the similarity matrix $W=\frac{|Z|+\left|Z^{\text{T}}\right|}{2}$ can be obtained to further construct the graph Laplacian and then run spectral clustering on that graph Laplacian to get the final clustering results.

2). Multi-View Subspace Clustering:

With multi-view information, each subspace representation Z_v can be obtained from each view. To get a consistent clustering result from multiple views, Yin et al. [30] shares the common coefficient matrix by enforcing the coefficient matrices from each pair of views as similar as possible. The optimization problem is formulated as

$$\{\begin{array}{c}\underset{Z^{(s)},s=1,2,\cdots ,m}{\min }{\sum }_{s=1}^m{\Vert X^{(s)}-X^{(s)}{Z}^{(s)}\Vert }_F^2\\ +\alpha {\sum }_{s=1}^m{\Vert Z^{(s)}\Vert }_1+\beta {\sum }_{1\le s\le t}{\Vert Z^{(s)}-Z^{(t)}\Vert }_1\\ s.t.\text{diag}\left(Z^{(s)}\right)=0,\forall s\in \{1,2,\cdots ,m\}.\end{array}$$ (15)

where ${\Vert Z^{(s)}-Z^{(t)}\Vert }_1$ is the l₁-norm based pairwise co-regularization constraint that can alleviate the noise problem. $\parallel Z{\parallel }_1$ is used to enforce sparse solution. diag(Z) denotes the diagonal elements of matrix Z, and the zero constraint is used to avoid trivial solution (each data point represents by itself).

Maria et al. [31] also considered the low rank and sparse representation to conduct multi-view subspace clustering. Wang et al. [32] enforced the similar idea to combine multi-view information. Apart from that, it adopted a multi-graph regularization with each graph Laplacian regularization characterizing the view-dependent non-linear local data similarity. At the same time, it assumes that the view-dependent representation is low rank and sparse and considers the sparse noise in the data. Wang et al. [33] proposed an angular based similarity to measure the correlation consensus in multiple views and obtained a robust subspace clustering for multi-view data. Zhang et al. [34] adopted linear correlation and neural networks to integrate the representation of each view and proposed two latent subspace MVC methods. To deal with the scenario where each view is unsufficient to discover the latent cluster structure, Huang et al. [35] proposed a multi-view intact subspace clustering by assuming a latent space and defining a mapping function from the latent space to view representation. Different from the above approaches, the three works [36], [37], [38] adopted general nonnegative matrix factorization formulation but shared a common representation matrix for the samples with both views and kept each view representation matrix specific. Zhao et al.[39] adopted a deep semi-nonnegative matrix factorization to perform MVC, and enforced a common coefficient matrix in the last layer to exploit the multi-view information. By introducing a label constraint matrix and enforcing representation matrix of each view close to a common one, Cai et al. [40] solved the MVC in semi-supervised settings.

The MVC methods based on a shared coefficient matrix are applied to multi-view subspace clustering, which assumes that the cluster structures can be found by identifying the low dimensional subspaces. This kind of MVC methods has great utility in the computer vision field. Typically, after the final low-dimensional representation is obtained, spectral clustering is conducted on the graph Laplacian constructed from that representation, so this group of methods possesses the same advantages and disadvantages as spectral clustering as discussed in Sec. III-A.

C. Common Indicator Matrix (Mainly Multi-View Nonnegative Matrix Factorization Clustering)

Nonnegative matrix factorization is commonly used in clustering. It enforces one of the factorized matrix as an indicator matrix whose nonzero entry can indicate which data point belongs to which cluster. Therefore, enforcing the indicator matrix for multiple views be same or similar is a natural way to conduct MVC.

1). Nonnegative Matrix Factorization (NMF):

For a nonnegative data matrix $X\in {ℝ}_{+}^{d\times N}$. Nonnegative Matrix Factorization [41] seeks two nonnegative matrix factors $U\in {ℝ}_{+}^{d\times K}$ and $V\in {ℝ}_{+}^{N\times K}$ such that their product is a good approximation of X:

$$X\approx U{V}^{\text{T}},$$ (16)

where K denotes the desired reduced dimension (for clustering, it is the number of clusters), U is the basis matrix, and V is the indicator matrix.

Due to the nonnegative constraints, a widely known property of NMF is that it can learn a part-based representation. It is intuitive and meaningful in many applications, such as in face recognition [41]. The samples in many of these applications e.g., information retrieval [41] and pattern recognition [42], [43] can be explained as additive combinations of nonnegative basis vectors. The NMF has been applied successfully to cluster analysis and has shown the state-of-the-art performance [41], [44].

2). MVC based on NMF:

To combine multi-view information in the NMF framework, Akata et al. [45] enforces a common indicator matrix in the NMF among different views to perform MVC. However, the indicator matrix $V^{(v)}$ might not be comparable at the same scale. In order to keep the clustering solutions across different views meaningful and comparable, Liu et al. [46] enforces a constraint to push each view-dependent indicator matrix towards a common indicator matrix, which leads to another normalization constraint inspired by the connection between NMF and probability latent semantic analysis. The final optimization problem is formulated as:

$$\{\begin{array}{c}\underset{U^{(v)},V^{(v)},v=1,2,\cdots ,m}{\min }{\sum }_{v=1}^m{\Vert X^{(v)}-U^{(v)}{V}^{(v)}\Vert }_F^2\\ +{\sum }_{v=1}^m{\Vert {\lambda }_v{V}^{(v)}-V^{\ast }\Vert }_F^2\\ s.t.\forall 1\le k\le K,{\Vert U_{.,k}^{(v)}\Vert }_1=1,U^{(v)},V^{(v)},V^{(\ast )}\ge 0.\end{array}$$ (17)

The constraint ${\Vert U_{.,k}^{(v)}\Vert }_1=1$ is used to guarantee V^(v) within the same range for different v such that the comparison between the view-dependent indicator matrix V^(v) and the consensus indicator matrix V^(∗) is reasonable. After obtaining the consensus matrix V^∗, the cluster label of data point i can be computed from $argma{x}_k{V}_{i,k}^{\ast }$.

3). Multi-View K-Means:

The k-means clustering method can be formulated using NMF by introducing an indicator matrix U. The NMF formulation of k-means clustering is

$$\{\begin{array}{l}\underset{U,V}{\min }{\Vert X^{\text{T}}-U{V}^{\text{T}}\Vert }_F^2\\ s.t.U_{i,k}\in \{0,1\},\sum _K^{k=1}{U}_{i,k}=1,\forall i=1,2,\cdots ,N\end{array}$$ (18)

where the columns of $V\in {ℝ}^{d\times K}$ give the cluster centroids.

Because the k-means algorithm has lower computational cost than those requiring eigen-decomposition, it can be a good choice for large scale data clustering. To deal with large scale multi-view data, Cai et al. [47] proposed a multi-view k-means clustering method by adopting a common indicator matrix across different views. The ${\mathcal{l}}_{2,1}$ norm has been applied in traditional NMF-based clustering methods with proved performance, such as model sparse and robustness. Herein the Frobenius norm has been replaced by a ${\mathcal{l}}_2$,1 norm, and different views are weighed differently according to their importance. The new optimization problem obtained from (18) is formulated as follows:

$$\{\begin{array}{c}\underset{V^{(v)},{\alpha }^{(v)},U}{\min }{\sum }_{v=1}^m{\left({\alpha }^{(v)}\right)}^{\gamma }{\Vert X^{{(v)}^{\text{T}}}-U{V}^{{(v)}^{\text{T}}}\Vert }_{2,1}\\ s.t.U_{i,k}\in \{0,1\},{\sum }_{k=1}^K{U}_{i,k}=1,{\sum }_{v=1}^m{\alpha }^{(v)}=1\end{array}$$ (19)

where ${\alpha }^{(v)}$ is the weight for the v-th view and γ is the parameter to control the weight distribution. By learning the weights α for different views, the important views will be emphasized.

Still based on multi-view k-means clustering (18), to deal with high dimensional problems in multiple views, Xu et al. [48] introduced one projection matrix for data of each view, and then conduct MVC by enforcing the common indicator matrix. Their optimization problem is formulated as

$$\{\begin{array}{c}\underset{V^{(v)},W^{(v)},U}{\min }{\sum }_{v=1}^m{\Vert X^{{(v)}^{\text{T}}}{W}^{(v)}-U{V}^{{(v)}^{\text{T}}}\Vert }_F\\ s.t.W^{{(v)}^{\text{T}}}{W}^{(v)}=I,U_{i,k}\in \{0,1\},{\sum }_{k=1}^K{U}_{i,k}=1.\end{array}$$ (20)

where $W^{(v)}\in {ℝ}^{D_v\times m_v}$ indicates the projection matrix which embeds the data matrix $X^{(v)}$ from $D_v$ to $m_v,m_v<D_v,\forall v$. Note that to deal with outliers, Frobenious norm (not squared) is adopted. By replacing Frobenious norm with a ${\mathcal{l}}_2$ norm and considering different importance of each view, a re-weighted discriminative embedding k-means method is formulated as

$$\{\begin{array}{l}\underset{V^{(v)},W^{(v)},{\alpha }^{(v)},U}{\min }{\sum }_{v=1}^m{\alpha }^{(v)}{\Vert X^{{(v)}^{\text{T}}}{W}^{(v)}-U{V}^{{(v)}^{\text{T}}}\Vert }_2\hfill \\ s.t.W^{{(v)}^{\text{T}}}{W}^{(v)}=I,U_{i,k}\in \{0,1\},{\sum }_{k=1}^K{U}_{i,k}=1.\hfill \end{array}$$ (21)

where ${\alpha }^{(v)}={\left(2{\Vert X^{{(v)}^{\text{T}}}{W}^{(v)}-U{V}^{{(v)}^{\text{T}}}\Vert }_F\right)}^{-1}$ is the weight for the vth view and is computed by current V^(v), W^(v), U.

Besides the above multi-view nonnegative matrix factorization clustering methods, Liu and Fu [49] introduced a categorical utility function to measure similarity between the common indicator matrix and the indicator matrix from each view and proposed a consensus based MVC method.

According to the work [50], when $W=H\ast H^T$ where W indicates similarity between data points or is a kernel, the above method is equivalent to spectral clustering or kernel k-means clustering. Although the single view methods (NMF, kernel k-means, and spectral clustering) have connections between each other, their multi-view versions are less connected because the views need to share some common factors, but there is only one factor H, which cannot be used in multiple of the views. However, for the multi-view k-means clustering method can be expressed as a multi-view NMF-based clustering problem with U indicating the indicator matrix according to formulation (18).

4). Others:

As mentioned earlier, there are generally two steps in subspace clustering: find a subspace representation and then run spectral clustering on the graph Laplacian computed from the subspace representation. To identify consistent clusters from different views, Gao et al. [51] merged these two steps in subspace clustering and enforced a common indicator matrix across different views. The formulation is given as follows:

$$\{\begin{array}{c}\underset{Z^{(v)},E^{(v)},U}{\min }{\sum }_{v=1}^m{\Vert X^{(v)}-X^{(v)}{Z}^{(v)}-E^{(v)}\Vert }_F^2\\ +{\lambda }_{1}tr\left(U^{\text{T}}\left(D^{(v)}-W^{(v)}\right)U\right)+{\lambda }_2{\sum }_{v=1}^m{\Vert E^{(v)}\Vert }_1\\ s.t.Z^{{(v)}^{\text{T}}},Z^{(v)}(i,i)=I,U^{\text{T}}U=I\end{array}$$ (22)

where Z^(v) is the subspace representation matrix of the v-th view, $W^{(v)}=\frac{\left|Z^{(v)}\right|+\left|Z^{{(v)}^{\text{T}}}\right|}{2}$, D^(v) is a diagonal matrix with diagonal elements defined as $d_{v_{i,i}}={\sum }_j{w}_{v_{i,j}}$, and U is the common indicator matrix which indicates a unique cluster assignment for all the views. Although this multi-view subspace clustering method is based on subspace clustering, it does not enforce a common coefficient matrix Z, but uses a common indicator matrix for different views. We thus categorize it into this group.

Wang et al. [52] integrates multi-view information via a common indicator matrix and simultaneously selects features for different data clusters by formulating the problem as follows:

$$\{\begin{array}{c}\underset{U^{\text{T}}U=I,W}{\min }{\Vert X^{\text{T}}W+{\mathbf{1}}_N{b}^{\text{T}}-U\Vert }_F\\ +{\gamma }_1\parallel W{\parallel }_{G_1}+{\gamma }_2\parallel W{\parallel }_{2,1}\end{array}$$ (23)

where $X=\left\{x_1,x_2,\cdots ,x_N\right\}\in {ℝ}^{d\times N}$, but here each x_i includes the features from all the m views and each view has d_j features such that $d={\sum }_{j=1}^m{d}_j$. The coefficient matrix $W=\left[w_1^1,\cdots ,w_K^1;\cdots ,\cdots ,\cdots ,;w_1^m,\cdots ,w_K^m\right]\in {ℝ}^{d\times K}$ contains the weights of each feature for K clusters, $b\in {ℝ}^{K\times 1}$ is the intercept vector, 1_N is N-element constant vector of ones, and $U={\left[u_1,\cdots ,u_N\right]}^{\text{T}}\in {ℝ}^{N\times K}$ is the cluster (assignment) indicator matrix. The regularizer $\parallel W{\parallel }_{G_1}={\sum }_{i=1}^K{\sum }_{j=1}^m{\Vert w_i^j\Vert }_2$ is the group $l_1$ regularization to evaluate the importance of an entire view’s features as a whole for a cluster whereas $\parallel W{\parallel }_{2,1}={\sum }_{i=1}^d{\Vert w^i\Vert }_2$ is the $l_{2,1}$ norm to select individual features from all views that are important for all clusters.

In [53], a matrix factorization approach was adopted to reconcile the clusters arisen from the individual views. Specifically, a matrix that contains the partition indicator of each individual view is created and then decomposed into two matrices: one showing the contribution of individual groupings to the final MVC, called meta-clusters, and the other showing the assignment of instances to the meta-clusters. Tang et al. [54] treated MVC as clustering with multiple graphs, each of which is approximated by matrix factorization with two factors: a graph-specific factor and a factor common to all graphs. Qian et al. [55] and Zong et al. [56] required each view’s indicator matrix to be as close as possible to a common indicator matrix and employed the Laplacian regularization to maintain the latent geometric structure of the views simultaneously. After learning indicator matrices of different views, Kang et al. [57] learned a common indicator matrix by measuring distance between indicator matrix and considering different impact each view enforces. Also by learning an indicator matrix and maximizing the worst-case performance against single-view case, Tao et al. proposed a reliable MVC method [58]. Zhang et al. [59] proposed a robust manifold matrix factorization to cluster hyperspectral images. Taking the discriminative information in low dimensional spaces into account, Ma et al. [60] extend the work [59] to multi-view clustering by enforcing the same indicator matrix.

Besides using a common indicator matrix, [61], [62], [63] introduced a weight matrix to indicate whether there are missing entries so that it can tackle the missing value problem. The multi-view self-paced clustering method [64] takes the complexities of the samples and views into consideration to alleviate the local minima problem. Tao et al. [65] enforces a common indicator matrix and seeks for the consensus clustering among all the views in an ensemble way. Another method that utilizes a common indicator matrix to combine multiple views [66] employed the linear discriminant analysis idea and automatically weighed different views. For graph-based clustering methods, the similarity matrix for each view is obtained, and then by minimizing the differences between a common indicator matrix and each similarity matrix, Nie et al. [67] provided one MVC method with multiple graphs.

The MVC methods that use a shared indicator matrix across views include the k-means or non-negative matrix factorization. On one side, it can scale to large scale datasets compared with spectral clustering based MVC approaches. On the other side, it can only be applied to data with cluster of spherical shape to cluster center. This is because k-means clustering makes a strong assumption that the data points assigned to a cluster are spherical about the cluster center.

D. Direct Combination (Mainly Multi-Kernel Based MVC)

Besides the methods that share some structure among different views, direct view combination via a kernel is another common approach to perform MVC. A natural way is to define a kernel for each view and then combine these kernels in a convex combination [68], [69], [70].

1). Kernel Functions and Kernel Combination Methods:

Kernel is a trick to learn nonlinear problem just by linear learning algorithm, since kernel function $K:\mathcal{X}\times \mathcal{X}\to ℝ$ can directly give the inner products in feature space without explicitly defining the nonlinear transformation φ. There are some common kernel functions as follows:

• Linear kernel: $K\left(x_i,x_j\right)=\left(x_i\cdot x_j\right)$,

• Polynomial kernel: $K\left(x_i,x_j\right)={\left(x_i\cdot x_j+1\right)}^d$,

• Gaussian kernel (Radial basis kernel): $K\left(x_i,x_j\right)=(\exp \left(-\frac{\Vert x_i-x_j\Vert {}^2}{2{\sigma }^2}\right)$,

• Sigmoid kernel: $K\left(x_i,x_j\right)=\left(\tanh \left(\eta x_i\cdot x_j+\nu \right)\right)$.

Kernel functions in a reproducing kernel Hilbert space (RKHS) can be viewed as similarity functions [71], [72] in a vector space, so we can use a kernel as a non-Euclidean similarity measure in the spectral clustering and kernel k-means methods. There have been some works on multi-kernel learning for clustering [73], [74], [75], [76], however, they are all for single-view clustering. If a kernel is derived from each view, and different kernels are combined elaborately to deal with the clustering problem, it will become the multi-kernel learning method for MVC. Obviously, multi-kernel learning [77], [78], [79], [80] can be considered as the most important part in this kind of MVC methods. There are three main categories of methods for combining multiple kernels [81]:

• Linear combination: It includes two basic subcategories: unweighted sum $K\left(x_i,x_j\right)={\sum }_{v=1}^m{k}_v\left(x_i^v,x_j^v\right)$ and weighted sum $K\left(x_i,x_j\right)={\sum }_{v=1}^m{w}_v^q{k}_v\left(x_i^v,x_j^v\right)$ where $w_v\in {ℝ}_{+}$ denotes the kernel weight for the vth view and ${\sum }_{v=1}^m{w}_v=1$ is the hyperparameter to control the distribution of the weights,

• Nonlinear combination: It uses a nonlinear function in terms of kernels, namely, multiplication, power, and exponentiation,

• Data-dependent combination: It assigns specific kernel weights for each data instance, which can identify the local distributions in the data and learn proper kernel combination rules for different regions.

2). Kernel K-Means and Spectral Clustering:

Kernel k-means [82] and spectral clustering [83] are two kernel-based clustering methods for optimizing the intra-cluster variance. Let $\varphi (\cdot ):x\in \mathcal{X}\to \mathcal{H}$ be a feature mapping which maps x onto a RKHS $\mathcal{H}$. The kernel k-means method is formulated as the following optimization problem,

$$\{\begin{array}{c}\underset{H}{\min }{\sum }_{i=1}^N{\sum }_{k=1}^K{H}_{ik}{\Vert \varphi \left(x_i\right)-{\mu }_k\Vert }_2^2\\ s.t.{\sum }_{k=1}^K{H}_{ik}=1,\end{array}$$ (24)

where $H\in {\{0,1\}}^{N\times K}$ is the cluster indicator matrix (also known as cluster assignment matrix), $n_k={\sum }_{i=1}^N{H}_{ik}$ and ${\mu }_k=\frac{1}{n_k}{\sum }_{i=1}^N{H}_{ik}\varphi \left(x_i\right)$ are the number of points in the kth cluster and the centroid of the kth cluster. With a kernel matrix K whose (i, j)th entry is $K_{ij}=\varphi {\left(x_i\right)}^{\text{T}}\varphi \left(x_j\right)$, $L=\text{diag}\left(\left[n_1^{-1},n_2^{-1},\cdots ,n_K^{-1}\right]\right)$ and $1_l\in {ℝ}^l$, a column vector of all ones, Eq. (24) can be equivalently rewritten as the following matrix-vector form,

$$\{\begin{array}{c}\underset{\mathit{H}}{\min }tr(K)-tr\left(L^{\frac{1}{2}}{H}^{\text{T}}KH{L}^{\frac{1}{2}}\right)\\ s.t.H{\mathbf{1}}_k={\mathbf{1}}_N.\end{array}$$ (25)

For the above kernel k-means matrix-factor form, the matrix H is binary, which makes the optimization problem difficult to solve. By relaxing the matrix H to take arbitrary real values, the above problem can be approximated. Specifically, by defining $U=H{L}^{\frac{1}{2}}$ and letting U take real values, further considering Tr(K) is constant, Eq. (25) will be relaxed to

$$\{\begin{array}{cc}\underset{U}{\max }& tr\left(U^{\text{T}}KU\right)\\ s.t.& U^{\text{T}}U=1_K.\end{array}$$ (26)

The fact $H^{\text{T}}H=L^{-1}$ leads to the orthogonality constraint on U which tells us that the optimal U can be obtained by the top K eigenvectors of the kernel matrix K. Therefore, Eq. (26) can be considered as the generalized optimization formulation of spectral clustering. Note that Eq. (26) is equivalent to Eq. (8) if the kernel matrix K takes the normalized Gram matrix form.

3). Multi-Kernel Based MVC:

Assume that there are m kernel matrices available, each of which corresponds to one view. To make a full use of all views, the weighted combination $K={\sum }_{v=1}^m{w}_v^p{K}^{(v)}$, $w_v^p\ge 0$, ${\sum }_{v=1}^m{w}_v^p=1$, $p\ge 1$ will be used in kernel k-means (26) and spectral clustering (8) to obtain the corresponding multi-view kernel k-means and multi-view spectral clustering [84]. Using the same nonlinear combination but specifically setting p = 1, Guo et al. [85] extended the spectral clustering to MVC with kernel alignment. Due to the potential redundance of the selected kernels, Liu et al. [86] introduced a matrix-induced regularization to reduce the redundancy and enhance the diversity of the selected kernels to attain the final goal of boosting the clustering performance. By replacing the original Euclidean norm metric in fuzzy c-means with a kernel-induced metric in the data space and adopting the weighted kernel combination, Zhang et al. [87] successfully extended the fuzzy c-means to MVC that is robust to noise and outliers. In the case when incomplete multi-view data set exists, by optimizing the alignment of the shared data instances, Shao et al. [88] collectively completes the kernel matrices of incomplete data sets. Liu et al. [89] integrated imputation and clustering into a unified learning procedure, but the computational and storage complexities of this method is quite high. To overcome these drawbacks, they proposed a late fusion method that effectively and efficiently conduct MVC with a three-step iterative procedure [90]. To overcome the cluster initialization problem associated with kernel k-means, Tzortzis et al. [91] proposed a global kernel k-means algorithm, a deterministic and incremental approach that adds one cluster in each stage, through a global search procedure consisting of several executions of kernel k-means from suitable initiations.

4). Others:

Besides multi-kernel based MVC, there are some other methods that use the direct combination of features to perform MVC like [66], [67], [92]. In [93], two-level weights: view wights and variable wights are assigned to the clustering algorithm for multi-view data to identify the importance of the corresponding views and variables. Zhou et al. [94] learns an optimal neighborhood Laplacian matrix by searching the neighborhood of both the linear combination of the first-order and high-order base Laplacian matrices simultaneously to conduct multi-view spectral clustering finally. To extend fuzzy clustering method to MVC, each view is weighted and the multi-view versions of fuzzy c-means and fuzzy k-means are obtained in [95] and [96], respectively.

Direct combination based MVC can adaptively tune the weights of each view, which is necessary and important when some views are of low quality. The consensus information among different views are not clear in the direct combination based MVC methods because there are no commonality shared between different views.

E. Combination After Projection (Mainly CCA-Based MVC)

For multi-view data with all views with the same data type, like categorical or continuous, it is reasonable to directly combine them together. However, in real-world applications, the multiple representations may have different data types, and it is difficult to compare them directly. For instance, in bioinformatics, genetic information can be one view while clinical symptoms can be another view in the cluster analysis of patients [97]. Obviously, the information cannot be combined directly. Moreover, high dimension and noise are also difficult to handle. To solve the above problems, the last yet important combination way is introduced: combination after projection. The most commonly used technique is Canonical Correlation Analysis (CCA) and the kernel version of CCA (KCCA).

1). CCA and KCCA:

To better understand this style of view combination, CCA and KCCA are briefly introduced (refer to [98] for more detail). Given two data sets $S_x=\left[x_1,x_2,\cdots ,x_N\right]\in {ℝ}^{d_x\times N}$ and $S_y=\left[y_1,y_2,\cdots ,y_N\right]\in {ℝ}^{d_y\times N}$ where each entry x or y has a zero mean, CCA aims to find a projection $w_x\in {ℝ}^{d_x}$ for x and another projection $w_y\in {ℝ}^{d_y}$ for y such that the correlation between the projection of S_x and S_y on w_x and w_y are maximized,

$$\rho =\underset{{\mathbf{w}}_{\mathbf{x}},{\mathbf{w}}_{\mathbf{y}}}{\max }\frac{{{\mathbf{w}}_{\mathbf{x}}}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}}{\sqrt{\left({{\mathbf{w}}_{\mathbf{x}}}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}\right)\left({{\mathbf{w}}_{\mathbf{y}}}^{\text{T}}{\mathbf{C}}_{\mathbf{y}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}\right)}}$$ (27)

where ρ is the correlation and ${\text{C}}_{\text{xy}}=\mathbb{E}\left[{\text{xy}}^{\text{T}}\right]$ denotes the covariance matrix of x and y with zero mean. Observing that ρ is not affected by scaling w_x or w_y either together or independently, CCA can be reformulated as

$$\{\begin{array}{c}\underset{{\mathbf{w}}_{\mathbf{x}},{\mathbf{w}}_{\mathbf{y}}}{\max }{{\mathbf{w}}_{\mathbf{x}}}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}\\ s.t.{\mathbf{w}}_{\mathbf{x}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}=1,\\ {\mathbf{w}}_{\mathbf{y}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{y}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}=1.\end{array}$$ (28)

which can be solved using the method of Lagrange multiplier. The two Lagrange multipliers λ_x and λ_y are equal to each other, that is ${\lambda }_x={\lambda }_y=\lambda$. If C_yy is invertible, w_y can be obtained as ${\mathbf{w}}_{\mathbf{y}}=\frac{1}{2}{\mathbf{C}}_{\mathbf{y}\mathbf{y}}{}^{-1}{\mathbf{C}}_{\mathbf{y}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}$ and ${\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\left({\mathbf{C}}_{\mathbf{y}\mathbf{y}}\right)}^{-1}{\mathbf{C}}_{\mathbf{y}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}={\lambda }^2{\mathbf{C}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}$. Hence, w_x can be obtained by solving an eigen problem. For different eigen values (from large to small), eigen vectors are obtained in a successive process.

The above canonical correlation problem can be transformed into a distance minimization problem. For ease of derivation, the successive formulation of the canonical correlation is replaced by the simultaneous formulation of the canonical correlation. Assume that the number of projections is p, the matrices W_x and W_y denote (w_x1, w_x2, . . . , w_xp) and (w_y1, w_y2, . . . , w_yp), respectively. The formulation that simultaneously identifies all the w’s can be written as an optimization problem with p iteration steps:

$$\{\begin{array}{c}\underset{\left({\mathbf{w}}_{\mathbf{x}1},{\mathbf{w}}_{\mathbf{x}2},\dots ,{\mathbf{w}}_{\mathbf{x}\mathbf{p}}\right),\left({\mathbf{w}}_{\mathbf{y}1},{\mathbf{w}}_{\mathbf{y}2},\dots ,{\mathbf{w}}_{\mathbf{y}\mathbf{p}}\right)}{\max }\sum _{i=1}^p{{\mathbf{w}}_{\mathbf{x}\mathbf{i}}}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}\mathbf{i}}\\ s.t.{w_{xi}}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}\mathbf{j}}=\{\begin{array}{cc}1& \text{if i}=\text{j,}\\ 0& \text{otherwise,}\end{array}\\ {w_{\mathbf{y}\mathbf{i}}}^{\text{T}}{\mathbf{C}}_{\mathbf{y}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}\mathbf{j}}=\{\begin{array}{cc}1& \text{if i}=\text{j,}\\ 0& \text{otherwise,}\end{array}\\ i,j=1,2,\dots ,p,\\ {{\text{w}}_{\text{xi}}}^{\text{T}}{\mathbf{C}}_{\text{xy}}{\text{w}}_{\text{yj}}=\mathbf{0},\\ i,j=1,2,\dots ,p,j\ne i.\end{array}$$ (29)

The matrix formulation to the optimization problem (29) is

$$\{\begin{array}{c}\underset{{\mathbf{w}}_{\mathbf{x}},{\mathbf{w}}_{\mathbf{y}}}{\max }Tr\left({\mathbf{w}}_{\mathbf{x}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}\right)\\ s.t.{\mathbf{w}}_{\mathbf{x}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}=\mathbf{I},\\ {\mathbf{w}}_{\mathbf{y}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{y}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}=\mathbf{I},\\ {\mathbf{w}}_{\mathbf{x}\mathbf{i}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{x}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}\mathbf{j}}=\mathbf{0},\\ {\mathbf{w}}_{\mathbf{y}\mathbf{i}}{}^{\text{T}}{\mathbf{C}}_{\mathbf{y}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}\mathbf{j}}=\mathbf{0},\\ i,j=1,\dots ,p,j\ne i.\end{array}$$ (30)

where I is an identity matrix with size p × p. Maximizing the objective function of Eq. (30) can be transformed into the equivalent form as follows:

$$\underset{{\mathbf{w}}_{\mathbf{x}},{\mathbf{w}}_{\mathbf{y}}}{\min }{\Vert {\mathbf{w}}_{\mathbf{x}}{}^{\text{T}}{\mathbf{S}}_{\mathbf{x}}-{\mathbf{w}}_{\mathbf{y}}{}^{\text{T}}{\mathbf{S}}_{\mathbf{y}}\Vert }_F,$$ (31)

which is used widely in many works [36], [38], [99].

KCCA uses the “kernel trick” to maximize the correlation between two non-linear projected variables. Analogous to Eq. (28), the optimization problem for KCCA is formulated as follows:

$$\{\begin{array}{c}\underset{{\mathbf{w}}_{\mathbf{x}},{\mathbf{w}}_{\mathbf{y}}}{\max }\frac{{\mathbf{w}}_{\mathbf{x}}^{\text{T}}{\mathbf{K}}_{\mathbf{x}}{\mathbf{K}}_{\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}}{\sqrt{\left({\mathbf{w}}_x{}^T{\mathbf{K}}_{\mathbf{x}}^2{\mathbf{w}}_{\mathbf{x}}\right)\left({\mathbf{w}}_y{}^T{\mathbf{K}}_{\mathbf{y}}^2{\mathbf{w}}_{\mathbf{y}}\right)}}\\ s.t.{\mathbf{w}}_{\mathbf{x}}^{\text{T}}{\mathbf{K}}_{\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}=\mathbf{1},\\ {\mathbf{w}}_{\mathbf{y}}^{\text{T}}{\mathbf{K}}_{\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}=1.\end{array}$$ (32)

In contrast to the linear CCA that works by solving an eigendecomposition of the covariance matrix, KCCA solves the following eigen-problem:

$$\left(\begin{array}{cc}0& K_x{K}_y\\ K_y{K}_x& 0\end{array}\right)\left(\begin{array}{l}{w}_x\hfill \\ w_y\hfill \end{array}\right)=\lambda \left(\begin{array}{cc}{K}_x^2& 0\\ 0& K_y^2\end{array}\right)\left(\begin{array}{l}{w}_x\hfill \\ w_y\hfill \end{array}\right).$$ (33)

2). CCA Based MVC:

Since cluster analysis in a high dimensional space is difficult, Chaudhuri et al. [100] first projects the data into a lower dimensional space via CCA and then clusters samples in the projected low dimensional space. Under the assumption that multiple views are uncorrelated given the cluster labels, it shows a weaker separation condition required to guarantee the algorithm successful. Blaschko et al. [101] projects the data onto the top directions obtained by the KCCA across different views and applies k-means to clustering the projected samples.

For the case of paired views with some class labels, CCA can still be applied by ignoring the class labels. However, the performance can be ineffective. To take an advantage of the class label information, Rasiwasia et al. [102] has proposed two solutions with CCA: mean-CCA and cluster-CCA. Consider two data sets each of which is divided into K different but corresponding classes or clusters. Given $S_x=\left\{x_1,x_2,\cdots ,x_K\right\}$ and $S_y=\left\{y_1,y_2,\cdots ,y_K\right\}$, where $x_k=\left\{x_1^k,x_2^k,\cdots ,x_{\left|x_k\right|}^k\right\}$ and $y_k=\left\{y_1^k,y_2^k,\cdots ,y_{\left|y_k\right|}^k\right\}$ are the data points in the kth cluster for the first and second views, respectively. The first solution is to establish correspondences between the mean cluster vectors in the two views. Given the cluster means $m_x^k=\frac{1}{\left|x_k\right|}{\sum }_{i=1}^{\left|x_k\right|}{x}_i^k$ and $m_y^k=\frac{1}{\left|y_k\right|}{\sum }_{i=1}^{\left|y_k\right|}{y}_i^k$, mean-CCA is formulated as

$$\rho =\underset{w_x,w_y}{\max }\frac{w_x{V}_{xy}{w}_y}{\sqrt{\left({\mathbf{w}}_{\mathbf{x}}{}^{\text{T}}{\mathbf{V}}_{\mathbf{x}\mathbf{x}}{\mathbf{w}}_{\mathbf{x}}\right)\left({\mathbf{w}}_{\mathbf{y}}{}^{\text{T}}{\mathbf{V}}_{\mathbf{y}\mathbf{y}}{\mathbf{w}}_{\mathbf{y}}\right)}},$$ (34)

where $V_{xy}=\frac{1}{K}{\sum }_{k=1}^K{m}_x^k{m}_y^{k^{\text{T}}}$, $V_{xx}=\frac{1}{K}{\sum }_{k=1}^K{m}_x^k{m}_x^{k^{\text{T}}}$ and $V_{yy}=\frac{1}{K}{\sum }_{k=1}^K{m}_y^k{m}_y^{k^{\text{T}}}$. The second solution is to establish a one-to-one correspondence between all pairs of data points in a given cluster across the two views of data sets and then standard CCA is used to learn the projections.

For multi-view data with at least one complete view (features for this view are available for all data points), Anusua et al. [103] borrowed the idea from Laplacian regularization to complete the incomplete kernel matrix and then applied KCCA to perform MVC. In another method for MVC, multiple data matrices $A^{(v)}\in {ℝ}^{N\times K_v}$, v = 1, 2, · · · , K each of which corresponds to a view are obtained in an intermediate step and then a consensus data matrix should be learned to approximate each view’s data matrix as much as possible. Due to the unsupervised property, however, the data matrices are often not directly comparable. Using the CCA formulation Eq. (31), Long et al. [104] projects one view’s data matrix first before comparing with another view’s data matrix.

The same idea can be used to tackle the incomplete view problem (i.e., there are no complete views). For instance, if there are only two views, the methods in [36], [38] split data into the portion of data with both views and the portion of data with only one view, and then projects each view’s data matrix so that it is close to the final indicator matrix. Multi-view information is connected by the common indicator matrix corresponding to the projected data from both views. Wang et al. [105] provides a MVC method using an extreme learning machine that maps the normalized feature space onto a higher dimensional feature space.

Combination after projection based MVC methods fit for scenarios where different views cannot be compared directly in original input space. Although the consensus information is used well in this group of MVC methods, the complementary information is not taken into account. This is contrary to direct combination based MVC approaches. Thus it is intriguing to explore whether it is possible to fuse this two groups of methods together to make full use of the consensus and complimentary information.

F. Other MVC methods

In subsections III-A, III-B, III-C, we have introduced three classes of similarity structure based MVC methods. In addition, there are also some methods to share other similar structures to perform MVC. By sharing an indicator vector across views in a singular value decomposition of multiple data matrices, Sun et al. [97], [106], [107] extend the biclustering [108] method to the multi-view settings. Wang et al. [109] chooses the Jaccard similarity to measure the cross-view clustering consistency and simultaneously considers the within-view clustering quality to cluster multi-view data. By sharing a shared subspace’s bidirectional sparsity, Fan et al. [110] proposed an MVC approach which can find an effective subspace dimension and deal with outliers simultaneously.

Apart from the above categorized methods, there are some other MVC methods. Different from exploiting the consensus information of multi-view data, Cao et al. [111] utilizes a Hilbert Schmidt Independence Criterion as a diversity term to explore the complementarity of multi-view information. It reduces the redundancy of multi-view information to improve the clustering performance. Based on the idea of “minimizing disagreement” between clusters from each view, De Sa [112] proposes a two-view spectral clustering that creates a bipartite graph of the views. Zhou et al. [113] defines a mixture of Markov chains on similarity graph of each view and generalize spectral clustering to multiple views. In [114], a transition probability matrix is constructed from each single view, and all these transition probability matrices are used to recover a shared low-rank transition probability matrix as a crucial input to the standard Markov chain method for clustering. By fusing the similarity data from different views, Lange et al. [115] formulates a nonnegative matrix factorization problem and adopts an entropy-based mechanism to control the weights of multi-view data. Zhu et al. [116] enforced a common affinity matrix to conduct MVC in one step. Liu et al. [117] chooses tensor to represent multi-view data and then performs cluster analysis via tensor methods. Based on an assumption that the exemplar of a cluster in one view is always an exemplar of that cluster in the other views, Zhang et al. [118] proposed a multi-view and multi-exemplar fuzzy clustering method which has a theoretical guarantee on the performance improvement compared with single-view clustering counterpart. In paper [119], via cross-view graph diffusion, a unified graph for multi-view data is learned to conduct final clustering.

IV. RELATIONSHIPS TO RELATED TOPICS

As we mentioned previously, MVC is a learning paradigm for cluster analysis with multi-view feature information. It is a basic task in machine learning and thus can be useful for various subsequent analyses. In machine learning and data mining fields, there are several closely related learning topics such as multi-view representation learning, ensemble clustering, multi-task clustering, and multi-view supervised and semi-supervised learning. In the following, we will elaborate the relationships between MVC and a few other topics.

A. Relationship to Multi-View Representation

Multi-view representation [120] is the problem of learning a more comprehensive or meaningful representation from multi-view data. According to [121], representation learning (also known as embedding learning or metric learning) is a way to take advantage of human ingenuity and prior knowledge to extract some useful but far-removed feature representation for the ultimate objective. Thus representation learning does not need to be unsupervised in nature. For instance, metric learning has mainly been studied from the supervised perspective, when class labels are present. Using the class labels, approaches usually form constraints, for example, pairwise or triplet-based constraints. Multi-view representation can be considered as a more basic task than MVC, since multi-view representation can be useful in broader purpose such as classification or clustering and so on. However, cluster analysis based on multi-view representation may not be ideal because the creation of multi-view representation is unaware of the final goal of clustering [122], [123].

In an archived survey article [120], multi-view representation methods are categorized into mainly two classes: the shallow methods and the deep methods. The shallow methods are mainly based on CCA, which may correspond to our subsection III-E. For the deep methods, there exist a large number of works [124], [125], [126], [127], [128], [129], [130] on multi-view representation. For multi-view deep clustering, there are also many recent works including [131], [132], [133], [134], [135], [136]. As mentioned above, the sequential way of first multi-view representation and then clustering is a natural way to perform MVC, but the ultimate performance is usually not good because of the gap in the two steps. Therefore, how to integrate clustering and multi-view representation learning into a simultaneous process is an intriguing direction up to date, especially for deep multi-view representation. In addition, although many MVC methods sprung up in recent years, it still has large space to develop, especially compared with the development of multi-view deep representation learning.

B. Relationship to Ensemble Clustering

Ensemble clustering [137] (also named consensus clustering or aggregation of clustering) is made up of two steps: generation step and consensus step. Generation step is used to generate several sets of clusterings of the data set while consensus step is used to combine those sets of clusterings to obtain a consensus clustering. MVC does not need to obtain the final clustering result based on the sets of clusterings from original datasets, the final clustering result can be directly obtained from original datasets. This is the big difference between ensemble clustering and MVC. Certainly, MVC can also conduct clustering from generation step and consensus step when original datasets are multi-view and those clusterings obtained in generation step are gotten from each view of the original datasets. Thus if ensemble clustering is applied to clustering with multiple views of data, it becomes a type of MVC method. In this sense, MVC and ensemble clustering have some overlaps. Therefore, some of the ensemble clustering techniques e.g., [138], [139], [140], [141], [142], [143] can be applied to MVC. These works [65], [144], [145] are representative multi-view ensemble clustering methods. Although the idea of ensemble clustering is simple, it has gained good performance in real-world application. Especially in many kaggle competitions held recently, ensemble mechanism is quite popular and performed well. Thus, more exploration in this direction can be done in future. However, it should be noted that MVC does not need to have clear separate generation and consensus steps. More works connect MVC and ensemble clustering can be investigated further.

C. Relationship to Multi-Task Clustering

Multi-task clustering improves the clustering performance of each task by transferring knowledge among the related tasks, such as in [146], [147], [148], [149], [150], [151]. Between MVC and multi-task clustering, there are two big differences: The first one is that multi-task cares about the performance of each task, while MVC just cares about a final consensus clustering performance not each view. The second one is that one works on multiple tasks while the other one works on multiple views. Multiple tasks can be based on multiple datasets, while multiple views have to be based on the same dataset (but just different views of this one dataset). If each task corresponds to clustering in a specific view of the same dataset, multiple clustering results will be obtained, and then ensemble clustering methods may be employed to fuse these clustering results. Therefore, multi-task clustering, potentially combined with ensemble clustering, can implement MVC in the scenario where each task corresponds to each view of the same data. In addition, multi-task clustering and MVC can be conducted simultaneously to improve the clustering performance [152], [153], [154]. However, we should still distinguish the differences between them, since multi-task clustering cares about the clustering performance of each task. Even if each task corresponds to each view of the dataset, multi-task clustering is still not equivalent to MVC. When multi-task clustering combines ensemble clustering further, it will achieve MVC. Thus some techniques and ideas in multitask clustering and ensemble clustering can be helpful for MVC

D. Relationship to Multi-View Supervised and Semi-Supervised Learning

The difference between MVC and multi-view supervised, semi-supervised learning lies in whether to use the label of the data. MVC does not use any label of the data while multi-view supervised learning [4], [155] uses the labeled data to learn classifiers (or other inference models), multi-view semi-supervised learning [3], [4] can learn classifiers with both the labeled and unlabeled data.

The commonality between them lies in the way to combine multiple views. Many widely recognized techniques for combining views in the supervised or semi-supervised settings, e.g., co-training [23], [156], co-regularization [157], [158], margin consistency [159], [160] can lend a hand to MVC if there is a mechanism to estimate the initial labels. Thus, the key point to conduct MVC with some techniques in multi-view supervised or semi-supervised learning is how to estimate the initial labels or get some pseudo labels to play the role of labels in multi-view supervised learning or multi-view semi-supervised learning.

V. APPLICATIONS

MVC has been successfully applied to various applications including computer vision, natural language processing, social multimedia, bioinformatics and health informatics and so on.

A. Computer Vision

MVC has been widely used in image categorization [30], [32], [51], [111], [161], [139], [162] and motion segmentation tasks [45], [163]. Typically, several feature types e.g., CEN-TRIST [164], ColorMoment [165], HOG [166], LBP [167] and SIFT [168] can be extracted from the images (see the Fig. 2 [51]) prior to cluster analysis. Yin et al. [30] proposed a pairwise sparse subspace representation for multi-view image clustering, which harnesses the prior information and maximizes the correlation between the representations of different views. Wang et al. [32] enforced between-view agreement in an iterative way to perform multi-view spectral clustering on images. Gao et al. [51] assumed a common low dimensional subspace representation for different views to reach the goal of MVC in computer vision applications. Cao et al. [111] adopted Hilbert Schmidt Independence Criterion as a diversity term to exploit the complementary information of different views and performed well on both image and video face clustering tasks. Jin et al. [161] utilized the CCA to perform multi-view image clustering for large-scale annotated image collections.

Fig. 2:

The five views (CENTRIST, ColorMoment, LBP, HOG and SIFT) on three sample images from Caltech101.

Ozay et al. [139] used consensus clustering to fuse image segmentations. Chi et al. [169] conducted MVC for web image retrieval ranking. Méndez et al. [162] adopted the ensemble way to perform MVC for MRI image segmentation. Nonnegative matrix factorization was adopted in [45] to perform MVC for motion segmentation. Djelouah et al. [163] addressed the motion segmentation problem by propagating segmentation coherence information in both space and time. Xin et al. [89] successfully applied MVC for person re-identification. Tao et al. [170] applied their proposed multi-view subspace clustering methods to background subtraction from multi-view videos.

B. Natural Language Processing

In natural language processing, text documents can be obtained in multiple languages. It is natural to use MVC to conduct document categorization [16], [17], [46], [51], [171], [172], [173] with each language as one view. Employing the co-training and co-regularization ideas, Kumar et al. [16], [17] proposed co-training MVC and co-regularization MVC, respectively. The performance comparison on multilingual data demonstrates the superiority of these two methods over single-view clustering. Liu et al. [46] extended nonnegative matrix factorization to multi-view settings for clustering multilingual documents. Kim et al. [171] obtained the clustering results from each view and then constructed a consistent data grouping by voting. Jiang et al. [172] proposed a collaborative PLSA method that combines individual PLSA models in different views and imports a regularizer to force the clustering results in an agreement across different views. Hussain [174] utilized an ensemble way to perform MVC on documents. Zhang et al. [43] adopted a MVC method with graph regularization to improve object recognition.

C. Social Multimedia

Currently, with the fast development of social multimedia, how to make full use of large quantities of social multimedia data is a challenging problem, especially when matching them to the “real-world concepts” such as the “social event detection”. Fig. 3 shows two such events: a concert and an NBA game. The pictures showed there form just one view, and other textural features such as tags and titles form the other view. Such a social event detection problem is a typical MVC problem. Petkos et al. [175] adopted a multi-view spectral clustering method to detect the social event and additionally utilized some known supervisory signals (the known clustering labels). Samangooei et al. [176] performed feature selection first before constructing the similarity matrix and applied a density based clustering to the fused similarity matrix. Petkos et al. [177] proposed a graph-based MVC to cluster the data from social multimedia. MVC has also been applied to grouping multimedia collections [178], news stories [179] and social web videos [180].

Fig. 3:

Some pictures from two social events: concerts (top row) and NBA game (bottom row).

D. Bioinformatics and Health Informatics

In order to identify genetic variants underlying the risk for substance dependence, Sun et al. [97], [106], [107] designed three multi-view co-clustering methods to refine diagnostic classification to better inform genetic association analyses. Chao et al. [181] extended the method in [97] to handle missing values that might appear in each view of the data, and used the method to analyze heroin treatment outcomes. The three views of data for heroin dependence patients are demonstrated in Fig. 4. Yu et al. [182], [183] designed a multi-kernel combination to fuse different views of information and showed superior performance on disease data sets. In [184], a MVC based on the Grassmann manifold was proposed to deal with gene detection for complex diseases. MVC is also applied to analyze athlete’s physical fitness test [185]. Recently, Rappoport and Shamir [186] provided a review on MVC on biomedical omics data sets.

Fig. 4:

Three views from health informatics: vital sign (left), urine drug screen (middle) and craving measure (right)).

VI. DATASETS AND EXPERIMENTS

To further analyze the advantages and disadvantages of each group of MVC algorithms, we provide several commonly used MVC datasets and conduct empirical evaluation to measure how each group of MVC algorithms performs.

A. Datasets

Six benchmark multi-view datasets are adopted, and the statistics of these datasets are summarized in Table I.

TABLE I:

Statistics of the multi-view datasets.

Dataset	# Samples	# Views	# Clusters	# Features in each view	Is entry non-negative
Sources	169	3	6	3560, 3631, 3068	No
Reuters	600	5	6	21526, 24892, 34121, 15487, 11539	Yes
Handwritten Digits	2000	3	10	76, 216, 64	Yes
COIL20	1440	3	20	30, 19, 30	Yes
YALE	165	3	15	4096, 3304, 6750	No
Movies	617	2	17	1878, 1398	No

3 Sources¹ is a news article dataset. These articles are collected from three news sources: BBC, Reuters, and Guardians. In original datasets, there are 948 articles that are reported by at least one of the three sources. Herein, 169 of these articles are included, and the bag-of-word representation is adopted to represent the articles. These 169 articles are dominated by one of the six topical classes: business, entertainment, health, politics, sport, technology.

Reuters [187] includes documents in five languages: English, French, German, Spanish, and Italian. These five language versions constructed five views of these documents, and bag of words is used to represent the features in each view. These documents belong to one of the six categories. 100 documents are randomly sampled from each category to construct a dataset of 600 documents.

Handwritten Digits is available from the UCI repository.² It has 2000 examples of handwritten digits (0–9) extracted from Dutch utility maps. There are 200 examples in each class, each represented with six feature sets. Following experiments in [188], three feature sets: 76 Fourier coefficients of the character shapes, 216 profile correlations and 64 Karhunen-Love Coefficiens are adopted.

COIL20 ³ consists of 1440 images belonging to 20 classes. Three views are represented by 30 isometric projection (ISO), 19 linear discriminant analysis (LDA), and 30 neighborhood preserving embedding (NPE), respectively.

YALE [189] consists of 165 images from 15 subjects, which has 11 images per subject and corresponds to different facial expressions or configurations. Each image is expressed by three heterogeneous feature sets with dimensions of 4096, 3304 and 6750.

Movies ⁴ includes 617 movies belonging to 17 genres. Each movie is described by two views: 1878 keywords and 1398 actors.

B. Compared Methods and Parameter Settings

In the experiment, six representative MVC algorithms corresponding to each group of MVC approaches are used to compare. To explore how deep MVC algorithms perform, one deep algorithm is chosen to compare. These algorithms are multi-view mixture-of-multinomials EM (MVMMEM) [10], co-regularization multi-view spectral clustering (Co-Reg) [17], multi-view low-rank sparse subspace clustering (MVLRSSC) [31], multi-view clustering via joint nonnegative matrix factorization (MultiNMF) [46], kernel-based weighted multi-view clustering (MVKKM) [84], multi-view clustering via canonical correlation analysis (MVCCA) [100], and multi-view clustering via deep matrix factorization (DeepNMF) [39].

As for the parameter settings, we try our best to set it according to that in their original papers. For MVMMEM, the number of rounds are selected from {5, 10, · · ·, 100}. For Co-Reg, parameter α is selected from 0.01 to 0.05 with step 0.01. For MVLRSSC, we tune the parameters β₁, β₂, λ^(v) according to [31]. For MultiNMF, λ_v is set to 0.01 for all views. According to MVKKM [84], good performance can be obtained with p = 1.5, we adopted this setting. For MVCCA, we kept vectors with canonical correlation bigger than 0.01. For DeepNMF, two layers with layer size [100,50] are designed for all the datasets except COIL20 ([18,9]), parameter β = 0.1, γ = 0.5.

To conduct a comprehensive evaluation, all the approaches are compared with six evaluation metrics: normalized mutual information (NMI), accuracy (ACC), adjusted rand index (ARI), F-score, Precision, and Recall. For all these metrics, the higher value indicates better clustering performance. All the algorithms are run 20 times and the mean and standard deviation of each metric is reported.

C. Experiment Results

The results are shown in Table II. On datasets 3 Sources, Reuters and Movies, MVLRSSC performs best. On datasets Handwritten Digits, COIL20, MVKKM outperforms all the other algorithms. On dataset YALE, DeepNMF obtained the best performance. These results are almost consistent on six different metrics except NMI on dataset Handwritten Digits and Recall on dataset Movies. On dataset Handwritten Digits, the performance in NMI for MVKKM and MVLRSSC are very close. On dataset Movies, although the performance in Recall for MVKKM is significantly better than that for MVLRSSC, the precision and comprehensive metric F1 score for MVKKM are worse.

TABLE II:

Performance of seven algorithms on six multi-view datasets. The mean and standard deviation of 20 runs of these algorithms are reported.“—” indicates that this dataset has negative entries, thus MultiNMF cannot apply.“/” indicates this algorithm only applies to two-view case directly but this dataset has more than two views. The best results among seven MVC algorithms on each dataset is shown in bold font.

Dataset	Method	ACC	F-score	Precision	Recall	NMI	ARI
Sources	MVMMEM	/	/	/	/	/	/
	Co-Reg	0.5434 (0.0097)	0.4648 (0.0100)	0.4985 (0.0124)	0.4373 (0.0100)	0.4894 (0.0086)	0.3161 (0.0131)
	MVLRSSC	0.6730 (0.0089)	0.6350 (0.0101)	0.6770 (0.0086)	0.6005 (0.0145)	0.5855 (0.0054)	0.5355 (0.0018)
	MultiNMF	0.4107 (0.0079)	0.3244 (0.0045)	0.2631 (0.0051)	0.4227 (0.0030)	0.3426 (0.0061)	0.0531 (0.0084)
	MVKKM	0.3550 (0)	0.3621 (0)	0.2298 (0)	0.8430 (0)	0.1131 (0)	−0.0064 (0)
	MVCCA	/	/	/	/	/	/
	DeepNMF	0.6509 (0.0076)	0.5079 (0.0047)	0.5645 (0.0060)	0.4614 (0.0065)	0.4892 (0.0107)	0.3780 (0.0056)
Reuters	MVMMEM	/	/	/	/	/	/
	Co-Reg	0.4800 (0.0068)	0.3699 (0.0030)	0.3386 (0.0041)	0.4091 (0.0047)	0.3000 (0.0038)	0.2308 (0.0041)
	MVLRSSC	0.5280 (0.0069)	0.4174 (0.0030)	0.3641 (0.0062)	0.4931 (0.0052)	0.3782 (0.0032)	0.2802 (0.0055)
	MultiNMF	—–	—–	—–	—–	—–	—–
	MVKKM	0.2283 (0)	0.2870 (0)	0.1737 (0)	0.8261 (0)	0.1909 (0)	0.0191 (0)
	MVCCA	/	/	/	/	/	/
	DeepNMF	0.2977 (0.0020)	0.2217 (0.0018)	0.2138 (0.0034)	0.2303 (0.0049)	0.1053 (0.0014)	0.0607 (0.0032)
Handwritten Digits	MVMMEM	/	/	/	/	/	/
	Co-Reg	0.7571 (0.0064)	0.6842 (0.0087)	0.6656 (0.0087)	0.7042 (0.0087)	0.7298 (0.0076)	0.6481 (0.0097)
	MVLRSSC	0.7699 (0.390)	0.7288 (0.0534)	0.6970 (0.0615)	0.7642 (0.0462)	0.7799 (0.0333)	0.6971 (0.0601)
	MultiNMF	—–	—–	—–	—–	—–	—–
	MVKKM	0.8650 (0)	0.7530 (0)	0.7411 (0)	0.7653 (0)	0.7740 (0)	0.7252 (0)
	MVCCA	/	/	/	/	/	/
	DeepNMF	0.7738 (0.0009)	0.7456 (0.0019)	0.7042 (0.0016)	0.7921 (0.0022)	0.7961 (0.0022)	0.7156 (0.0021)
COIL20	MVMMEM	/	/	/	/	/	/
	Co-Reg	0.9591 (0.0146)	0.9643 (0.0129)	0.9436 (0.0199)	0.9871 (0.0050)	0.9899 (0.0037)	0.9623 (0.0136)
	MVLRSSC	0.9767 (0.0078)	0.9799 (0.0065)	0.9686 (0.0099)	0.9922 (0.0028)	0.9943 (0.0019)	0.9788 (0.0068)
	MultiNMF	—–	—–	—–	—–	—–	—–
	MVKKM	1 (0)	1 (0)	1 (0)	1 (0)	1 (0)	1 (0)
	MVCCA	/	/	/	/	/	/
	DeepNMF	0.3857 (0.0050)	0.2688 (0.0121)	0.2133 (0.0163)	0.3651 (0.0156)	0.5144 (0.0029)	0.2202 (0.0142)
YALE	MVMMEM	/	/	/	/	/	/
	Co-Reg	0.5913 (0.0140)	0.4599 (0.0150)	0.4376 (0.0149)	0.4851 (0.0155)	0.6418 (0.0113)	0.4229 (0.0161)
	MVLRSSC	0.5677 (0.0103)	0.4141 (0.0090)	0.3939 (0.0090)	0.4368 (0.0093)	0.6088 (0.0082)	0.3739 (0.0097)
	MultiNMF	0.5188 (0.0054)	0.3652 (0.0149)	0.3470 (0.0111)	0.3855 (0.0201)	0.5602 (0.0203)	0.3217 (0.0154)
	MVKKM	0.6364 (0)	0.4732 (0)	0.4064 (0)	0.5661 (0)	0.6855 (0)	0.4329 (0)
	MVCCA	/	/	/	/	/	/
	DeepNMF	0.7446(0.0191)	0.5664 (0.0138)	0.5522 (0.0168)	0.5815 (0.0107)	0.7312 (0.0103)	0.5375 (0.0149)
Movies	MVMMEM	0.2592 (0.0163)	0.1538 (0.0134)	0.1460 (0.0135)	0.1627 (0.0150)	0.2529 (0.0144)	0.0955 (0.0144)
	Co-Reg	0.2615 (0.0033)	0.1517 (0.0023)	0.1402 (0.0022)	0.1657 (0.0031)	0.2657 (0.0031)	0.0916 (0.0024)
	MVLRSSC	0.3180 (0.0053)	0.1933 (0.0034)	0.1913 (0.0032)	0.1955 (0.0035)	0.3184 (0.0029)	0.1403 (0.0036)
	MultiNMF	0.1900 (0.0130)	0.1220 (0.0055)	0.0722 (0.0038)	0.3966 (0.0412)	0.2073 (0.0115)	0.0208 (0.0068)
	MVKKM	0.1005 (0)	0.1146 (0)	0.0611 (0)	0.9241 (0)	0.0670 (0)	(0)
	MVCCA	0.1295 (0.0036)	0.0607 (0.0013)	0.06175 (0.0014)	0.0596 (0.0014)	0.0854 (0.0058)	0.0007 (0.0015)
	DeepNMF	0.1847 (0.0033)	0.0945 (0.0028)	0.0911 (0.0035)	0.0981 (0.0026)	0.1626 (0.0027)	0.0332 (0.0035)

Datasets 3 Sources, Reuters and Movies consist of text information. Due to special topic properties, low rank and sparsity are important when conducting multi-view clustering, thus the algorithm MVLRSSC that take low rank and sparsity into account performs well. Datasets Handwritten Digits, COIL20 and YALE are datasets of images, maybe it is necessary to use nonlinear or deep structure to learn the abstract or meaningful clusters, thus MVKKM and DeepNMF perform better on these datasets. In addition, different views contribute different in final clustering, thus different weights should be given them to promote the performance, thus MVKKM can be a good choice. From the results, we can find that MultiNMF just applied to datasets 3 Sources, YALE and Movies, that is because MultiNMF can apply to the scenario where all entries of the datasets are non-negative. This is one limitation of MultiNMF. Results in Table II also shows that algorithms MVMMEM and MVCCA just apply to dataset Movies, that is because they are suitable for the datasets with only two views, thus this is the limitation of these algorithms. MVMMEM performs worse than MVLRSSC, as well as Co-Reg, better than MultiNMF, MVKKM, MVCCA and DeepNMF. It can be seen that generative algorithms has the potential to be comparable with state of the art discriminative algorithms. Although MultiNMF has non-negative property, it does not perform well compared with other group of algorithms on the above datasets. This maybe because compared with non-negative property, other properties like low rank, sparsity, weights difference are more important for performance improvement.

Based on the results in Table II, we can find that multi-view subspace clustering group, multi-kernel MVC, and deep MVC algorithms perform well. Spectral clustering based MVC, NMF based MVC and MVCCA perform worse than the above algorithms on the six commonly used datasets. Generative MVC performs better than many discriminative ones, thus it is worth attracting more attention in future. In this experiment, we focused on clustering performance, a more comprehensive study including time cost factor, and more advanced MVC algorithms that are worth further exploration.

VII. OPEN PROBLEMS

We have identified several problems that are still underexplored in the current body of MVC literature. We discuss these problems in this section.

A. Large Scale Problem (size and dimension)

In modern life, large quantities of data are generated every day. For instance, several million posts are shared per minute in Facebook, which include multiple data forms (views): videos, images and texts. At the same time, a large amount of news are reported in different languages, which can also be considered as multi-view data with each language as one view. However, most of the existing MVC methods can only deal with small datasets. It is important to extend these methods to large scale applications. For instance, it is difficult for the existing multi-view spectral clustering based methods to work on datasets of massive samples due to the expensive computation of graph construction and eigen-decomposition. Although some previous works such as [190], [191], [192], [193] attempted to accelerate the spectral clustering method to scale with big data, it is intriguing to extend them effectively to the multi-view settings. Recently, Zhang et al. [194] proposed an interesting idea to solve large scale problem by encoding multi-view image data into a compact common binary code space and then conduct binary clustering.

Another type of big data has high dimensionality. There is a large quantity of single-view clustering methods [195] to deal with this kind of problem, However, there is still one special class of such problem tough to deal with. For instance, in bioinformatics, each person has millions of genetic variants as genetic features where, compared with the problem dimension, the number of samples is low. Using genetic features in a clinical analysis with another view of clinical phenotypes often forms a multi-view analytics problem. How to deal with such a clustering problems is tough due to the over-fitting problem. Although feature selection [196], [197] or feature dimension reduction [198] like PCA is commonly used to alleviate this problem in single-view settings, there are no convincing methods up to now, especially because deep learning cannot cope with it due to the properties: small size and high feature dimension. It may recall new theory to appear to handle this problem.

B. Incomplete Views or Missing Value

MVC has been successfully applied to many applications as shown in Section V. However, there is an underlying problem hidden behind: what if one or more views are incomplete? This is very common in real-world applications. For example, in multi-lingual documents, many documents may have only one or two language versions; in social multimedia, some sample may miss visual or audio information due to sensor failure; in health informatics, some patients may not take certain lab tests to cause missing views or missing values. Some data entries may be missing at random while others are non-random [181]. Simply replacing the missing entries with zero or mean values [199] is a common way to deal with the missing value problem, and multiple imputation [200] is also a popular method in statistical field. The missing entries can be generated by the recently popular generative adversarial networks [201]. However, without considering the differences of random and non-random effects in missing data, the clustering performance is not ideal [181].

Up to now, there have already been several multi-view works [202], [36], [37], [88], [38], [61], [63], [103] that attempted to solve the incomplete view problem. Two methods in [61], [63] introduced a weight matrix M_i,j to indicate whether the ith instance present in the jth view. For the two-view case, the method in [36] reorganized the multi-view data to include three parts: samples with both views, samples only having view 1 and samples only having view 2 and then analyzed them to handle missing entries. Assuming that there is at least one complete view, Trivedi et al. [103] used the graph Laplacian to complete the kernel matrix with missing values based on the kernel matrix computed from the complete view. Shao [88] borrowed the same idea to deal with multi-view setting. Instead imputing kernel matrix, Liu [203] imputed each base matrix generated by incomplete views with a learned consensus clustering matrix. It is noted that all these methods deal with incomplete views or missing value with some constraints, but they do not aim to deal with the situation with arbitrarily missing values in any of the views. In other words, this situation is that all views have missing values and the samples just miss a few features in a view. Obviously, the above methods have significant limitations that cannot make full use of the available multi-view incomplete information. In addition, all existing methods do not take into consideration the difference between random and non-random missing patterns. Therefore, it is worth exploring how to use the mixed types of data in multi-view analysis.

C. Initialization and Local Minima

For MVC methods based on k-means, the initial clusters are very important and different initalizations may lead to different clustering results. It is still challenging to select the initial clusters effectively in MVC and even in single-view clustering settings.

Most NMF-based methods rely on non-convex optimization formulations, and thus are prone to the local optimum problem, especially when missing values and outliers exist. By enforcing a consistent clustering result in different view, Zhao et al. [173] formulated a jointly convex optimization formulation and additionally using some side information. Self-paced learning [204] is a possible solution, and Xu et al. [64] applied it to MVC to alleviate the local minimum problem.

The generative convex clustering method [8] is an interesting approach to avoid the local minimum problem. In [12], a multi-view version of the method in [8] is proposed and shows good performance. This kind of generative methods may be another good direction worth further exploring.

D. Deep Learning

Recently, Deep learning has demonstrated outstanding performance in many applications such as speech recognition, image segmentation, object detection and so on. However, compared with the fast growth of supervised deep learning and unsupervised deep representation learning, deep clustering still has a lot of room to develop, especially multi-view deep clustering. A natural way to conduct deep clustering or multi-view deep clustering is to conduct clustering on the representation obtained from single-view representation learning or multi-view representation learning. In fact, there should be many advanced ways to explore how to conduct multi-view deep clustering.

Recently, there indeed appeared a number of deep clustering works. For example, the works in [205], [206], [207] borrowed the supervised deep learning idea to perform supervised clustering. In fact, they can be considered as performing semi-supervised learning. So far, there are already several truly deep clustering works [131], [132], [208]. Tian et al. [131] proposed a deep clustering algorithm that is based on spectral clustering, but replaced eigenvalue decomposition by a deep auto-encoder. Xie et al. [132] proposed a clustering approach using deep neural network which can learn representation and perform clustering simultaneously. It is interesting to explore how to extend them to multi-view scenarios.

Besides deep clustering works, there also exist some MVC methods. Huang et al. [208] proposed to use multiple layer matrix factorization and shared the same representation matrix across different views to conduct MVC. Experimental results demonstrates the superiority of this deep learning methods to multi-view shallow clustering methods like co-training clustering, co-regularization clustering and multi-view k-means clustering. By using auto-encoder architecture, Zhu et al [135] designed a diverse net and universal net to make full use of the complementary and consensus information among multiple views to implement MVC. To let clustering label to guide the representation learning, Sun et al. [136] proposed another deep subspace MVC method in a semi-supervised way. Li et al. [133] presented a deep MVC approach borrowing the idea and architecture of Generative Discriminative Network (GAN). Inspired by the great success obtained by using attention mechanism in deep learning fields, Zhou et al. [134] explored a MVC method by combing GAN and attention mechanisms, and experiments support its effectiveness.

Compared with traditional multi-view shallow clustering methods, the aforementioned multi-view deep clustering methods demonstrated better performance due to several reasons. Firstly, deep networks adopted in multi-view deep clustering methods have better expression ability, maybe it can discover the more real structure of the multi-view data. Secondly, part of them adopt end-to-end multi-view deep clustering way. The representation obtained amid can reflect multi-view data comprehensively and, at the same time, serve to the final goal clustering well. However, there are still large space to explore and develop in this direction. Firstly, there are more and more novel deep learning architectures; how to extend them to multi-view scenarios needs more investigation. Secondly, although some end-to-end multi-view deep clustering methods appeared, more such methods are expected, since multi-view deep representation learning developed more sufficiently than multi-view deep clustering, and it is simple and natural to run clustering algorithm on the representation obtained from multi-view deep representation learning. However, the separate process to deal with multi-view deep clustering has its limitations, like being unaware of clustering goal in representation learning. Thirdly, deep learning techniques has its special properties; more ways to combine multiple views can be designed to serve to multi-view deep clustering. Fourth, Some theoretical investigation should be conducted to unfold how and why multi-view deep clustering shows better performance than traditional shallow methods.

E. Mixed Data Types

Multi-view data may not necessarily just contain numerical or categorical features. They can also have other types such as symbolic, ordinal, etc. These different types can appear simultaneously in the same view, or in different views. How to integrate different types of data to perform MVC is worthy of careful investigation. Converting all of them to categorical type is a straightforward solution. However, much information will be lost during such processing. For example, the difference of the continuous values categorized into the same category is ignored. Paper [209] proposed a solution to mixed data type problem with vine copulas. It is worth more exploring to make full use of the information within mixed data types in MVC settings.

F. Multiple Solutions

Most of the existing MVC, even single-view clustering, algorithms only output a single clustering solution. However, in real-world applications, data can often be grouped in many different ways, and all these solutions are reasonable and interesting from different perspectives. For example, it is both reasonable to group the fruits apple, banana, and grape according to the fruit type or color. Until now, to the best of our knowledge, there are very few works along this direction [210], [211], [212]. Cui et al. [210] proposed to partition multi-view data by projecting the data to a space that is orthogonal to the current solution so that multiple non-redundant solutions were obtained. In another work [211], Hilbert-Schmidt Independence Criterion was adopted to measure the dependence across different views and then one clustering solution was found in each view. Chang et al. [212] automatically learned multiple expert views and the clustering structure corresponding to each view in a Bayesian probabilistic model. MVC algorithms that can produce multiple solutions should attract more attention in the future.

VIII. CONCLUSION

To sort out existing MVC methods, we proposed a novel taxonomy to introduce them. Similar to machine learning method categorizations, we split MVC methods into two classes: generative methods and discriminative methods. Based on the way to combine multiple views, discriminative methods are further split into five main classes, the first three of which have a commonality: sharing certain structures across the views. The fourth one uses direct combinations of the views, while the fifth one employs view combinations after projections. Compared with discriminative methods, generative methods have developed far less sufficiently. Although it has inherent limitation, it can deal with missing data and get global optima easily, thus it calls for more attention. To better understand MVC, we elaborate on the relationships between MVC and several closely-related learning topics. We have also introduced several real-world applications of MVC and, most importantly, we conducted a comprehensive experimental study on representative MVC algorithms of each group to further analyze the advantages and disadvantages of them, and finally pointed out some interesting and challenging directions to guide researchers to advance in future.

Impact Statement—

Multi-view clustering (MVC) is a substantial extension of clustering with the ability to make full use of the multi-view information of objects. While it is true that one can cluster objects by simply concatenating the multiple views, the correlations between those views are not explicitly modeled and could bias the resultant clusters.

To improve the performance of clustering by leveraging the multi-view information, this paper provides a systematic survey of the existing MVC approaches. We further present an experimental comparison of the representative algorithms from each category. Our most significant findings include: multi-view subspace clustering, multi-kernel MVC, deep MVC algorithms, and generative MVC methods perform better than other discriminative ones. We suggest more research is needed on generative MVC. Moreover, some important future directions such as incomplete MVC and deep MVC are proposed.

Biography

Guoqing Chao received the B.S. degree from Xinyang Normal University, Xinyang, China, in 2009. He got the Ph.D. degree with the Department of Computer Science and Technology, East China Normal University, Shanghai, China in 2015. After that, he was a PostDoc in University of Connecticut and Northwestern University, US and Singapore Management University, Singapore. Currently, he works with Harbin Institute of Technology at Weihai, China. His research interests include machine learning, data mining and bioinformatics.

Shiliang Sun is a professor at the School of Computer Science and Technology and the head of the Pattern Recognition and Machine Learning Research Group, East China Normal University. He received the B.E. degree from Beijing University of Aeronautics and Astronautics in 2002, and the Ph.D. degree from Tsinghua University in 2007. He is a member of the PASCAL network of excellence, and on the editorial boards of multiple international journals. His research interests include approximate inference, learning theory, sequential modeling, kernel methods, and their applications.

Jinbo Bi, Professor, received a Ph.D. degree in mathematics from Rensselaer Polytechnic Institute, USA, and a master degree in Electrical Engineering and Automatic Control from Beijing Institute of Technology, China. She is a professor of Computer Science and Engineering at the University of Connecticut. Prior to her current appointment, she worked with Siemens Medical Solutions on computer aided diagnosis research and Partners Healthcare on clinical decision support systems, respectively. Her research interests include machine learning, data mining, bioinformatics and biomedical informatics.