Comments on: Probability Enhanced Effective Dimension Reduction for Classifying Sparse Functional Data

The authors are to be congratulated for their solid contribution in providing a powerful method that handles classification and dimension reduction problems with functional data sets. This type of problems has drawn much attention in the literature, and is known to be difficult due to the complex structure of the corresponding data sets. In particular, many existing dimension reduction methods ignore the relationship between predictors and labels, and perform dimension reduction only using the covariates. Such procedures can be suboptimal and may lead to unstable results, especially when the predictors are sparsely observed. The proposed PEFCS method integrates the observed labels in the dimension reduction step by estimating class conditional probabilities, and is shown to enjoy more competitive and robust performance in numerical examples.

This interesting paper leads to many promising research directions. For example, class conditional probability (we denote it by $P_j\left({\widehat{X}}_i\right)=\text{pr}\left(Y=j|{\widehat{X}}_i\right))$ estimation is a crucial step in the PEFCS method. In the literature, it is known that classification methods can be grouped into two main categories: soft and hard classifiers (Wahba, 2002; Liu et al, 2011). Soft classifiers directly estimate class conditional probabilities, which further leads to classification rules. Typical examples of soft classification include Fisher’s LDA and logistic regression. In contrast, hard classifiers bypass direct estimation of probabilities and focus on classification boundary estimation. Typical examples of hard classification include the support vector machine (SVM, Boser et al, 1992; Cortes and Vapnik, 1995) and ψ-learning (Shen et al, 2003). In Liu et al (2011), it was observed that the classification performance of various classifiers depends heavily on the underlying distribution of (X, Y). The authors use the hinge loss for the SVM in this paper. Therefore, a possible generalization of the proposed technique is to employ a more general loss function in their equation (5) for probability estimation, instead of using the weighted SVM. We will briefly discuss the idea below.

Consider the optimization problem

$$\underset{g\in {\mathit{ℱ}}_K}{min}{\displaystyle \sum _{i=1}^n\mathit{\ell }\left\{Y_{i}g\left({\widehat{X}}_i\right)\right\}}+\mathrm{\lambda }{\Vert g\Vert }_{{\mathrm{ℱ}}_K}^2,$$ (1)

where ℓ(·) is a differentiable loss function for a soft classifier. One can verify that $P_{+1}\left({\widehat{X}}_i\right)$ can be estimated using ${\mathit{\ell }}^{\prime }\left\{-\widehat{g}\left({\widehat{X}}_i\right)\right\}/{\mathit{\ell }}^{\prime }\left\{\widehat{g}\left({\widehat{X}}_i\right)\right\}$ (Liu et al, 2011). For standard classification where the predictors are scalars or vectors, Liu et al (2011) pointed out that when the underlying class conditional probability, as a function of the predictors, is relatively smooth, soft classifiers tend to perform better than the hard ones. Moreover, the transition behavior from soft to hard classifiers were thoroughly investigated using the large-margin unified machine family proposed by Liu et al (2011). For functional data classification, the comparison between soft and hard classifiers and the corresponding transition behavior are largely unknown, and further exploration in this direction can be very interesting.

Another potential research topic is to extend the PEFCS methodology to handle multicategory problems. In this case, the construction of slices in the EDR method becomes more involved. In particular, when Y ∈ {+1, −1}, only one direction of the EDR space can be recovered, because of the existence of homogeneity in learning problems with binary responses. To overcome this difficulty, Shin et al (2014) proposed to construct slices based on $P_{+1}\left({\widehat{X}}_i\right)$. In multicategory classification, estimation of the class conditional probabilities becomes more complex, as one needs to estimate a probability vector $\left\{P_1\left({\widehat{X}}_i\right),P_2\left({\widehat{X}}_i\right),\dots ,P_k\left({\widehat{X}}_i\right)\right\}$. Furthermore, how to construct $S_{\left(P_1,P_2,\dots ,P_k\right)|\mathit{X}}$ remains unclear. Therefore, it can be interesting and challenging to develop new methodology in this future research direction. Next, we provide one possible way to generalize the PEFCS methodology for multicategory problems.

For margin-based classification, when the number of classes is three or larger, one classification function g(·) is not enough to discriminate all classes. To overcome this difficulty, a common approach in the literature is to use k functions for k classes, and impose a sum-to-zero constraint on the k functions to reduce the parameter space and to ensure some theoretical properties such as Fisher consistency. Recently, Zhang and Liu (2014) suggested that using k functions and the sum-to-zero constraint can be inefficient and suboptimal, and proposed the angle-based large margin classifiers for multicategory classification. In particular, consider a simplex W with k vertices {W₁,…, W_k} in a (k − 1)-dimensional space, such that

$${\mathit{W}}_j=\{\begin{array}{ll}{\left(k-1\right)}^{-1/2}{\mathbf{1}}_{k-1},& j=1,\\ -\left(1+k^{1/2}\right)/\left\{{\left(k-1\right)}^{3/2}\right\}{\mathbf{1}}_{k-1}+{\left\{k/\left(k-1\right)\right\}}^{1/2}{\mathit{e}}_{j-1},& 2\le j\le k,\end{array}$$

where 1_k−1 is a vector of 1’s with length k − 1, and e_j ∈ ℝ^k−1is a vector with the jth element 1 and 0 elsewhere. In angle-based classification, one uses a (k − 1)-dimensional classification function f = (f₁,…, f_k−1)^T, which maps x to f(x) ∈ ℝ^k−1, where x is the predictor vector. Observe that f introduces k angles with respect to W₁,…, W_k, namely, ∠(f, W_j); j = 1,…, k. The prediction rule is based on which angle is the smallest. In particular, ∠(x) = argmin_j∈{1,…,_k} ∠(f, W_j), where ∠(x) is the predicted label for x. Based on the observation that

$$\underset{j\in \left\{1,\dots ,k\right\}}{\text{arg}min}\angle \left(\mathit{f},W_j\right)=\underset{j\in \left\{1,\dots ,k\right\}}{\text{arg}max}\langle \mathit{f},{\mathit{W}}_j\rangle ,$$

Zhang and Liu (2014) proposed the following optimization problem for the angle-based classifier

$$min{\displaystyle \sum _{i=1}^n\mathit{\ell }\left\{\langle {\mathit{W}}_{y_i},\mathit{f}\left({\mathit{x}}_i\right)\rangle \right\}+\mathrm{\lambda }\mathit{J}\left(\mathit{f}\right),}$$ (2)

where ℓ(·) is a binary margin-based surrogate loss function, J(f) is a penalty on f to prevent overfitting, and λ is a tuning parameter to balance the goodness of fit and the model complexity. One advantage of the angle-based classifier is that it is free of the commonly used sum-to-zero constraint, hence it can be more efficient for learning with big data sets. Thus, generalization of the PEFCS method in the angle-based framework should be feasible and promising.