Bayes Consistency vs. $\mathcal{H}$-Consistency: The Interplay between Surrogate Loss Functions and the Scoring Function Class

Abstract

A fundamental question in multiclass classification concerns understanding the consistency properties of surrogate risk minimization algorithms, which minimize a (often convex) surrogate to the multiclass 0–1 loss. In particular, the framework of calibrated surrogates has played an important role in analyzing Bayes consistency of such algorithms, i.e. in studying convergence to a Bayes optimal classifier (Zhang, 2004; Tewari and Bartlett, 2007). However, follow-up work has suggested this framework can be of limited value when studying $\mathcal{H}$-consistency; in particular, concerns have been raised that even when the data comes from an underlying linear model, minimizing certain convex calibrated surrogates over linear scoring functions fails to recover the true model (Long and Servedio, 2013). In this paper, we investigate this apparent conundrum. We find that while some calibrated surrogates can indeed fail to provide $\mathcal{H}$-consistency when minimized over a naturallooking but naïvely chosen scoring function class $\mathcal{F}$, the situation can potentially be remedied by minimizing them over a more carefully chosen class of scoring functions $\mathcal{F}$. In particular, for the popular one-vs-all hinge and logistic surrogates, both of which are calibrated (and therefore provide Bayes consistency) under realizable models, but were previously shown to pose problems for realizable $\mathcal{H}$-consistency, we derive a form of scoring function class $\mathcal{F}$ that enables Hconsistency. When $\mathcal{H}$ is the class of linear models, the class $\mathcal{F}$ consists of certain piecewise linear scoring functions that are characterized by the same number of parameters as in the linear case, and minimization over which can be performed using an adaptation of the min-pooling idea from neural network training. Our experiments confirm that the one-vs-all surrogates, when trained over this class of nonlinear scoring functions $\mathcal{F}$, yield better linear multiclass classifiers than when trained over standard linear scoring functions.

1. Introduction and Background

Consider a standard multiclass classification problem, with instance space $\mathcal{X}\subseteq {ℝ}^d$, label space $\mathcal{Y}=[n]:=\{1,\dots ,n\}$ with n > 2 classes, and standard 0–1 loss ${\ell }_{0-1}:\mathcal{Y}\times \mathcal{Y}\to {ℝ}_{+}$ given by ${\ell }_{0-1}(y,\widehat{y})=1(\widehat{y}\ne y)$. There is an unknown probability distribution D on $\mathcal{X}\times \mathcal{Y}$; given a training sample S = ((x₁, y₁), …, (x_m, y_m)) containing examples drawn i.i.d. from D, the goal is to learn a classifier $h:\mathcal{X}\to \mathcal{Y}$ with small 0–1 generalization error on new examples drawn from D:

$${\mathrm{er}}_D^{0-1}[h]={\mathbf{E}}_{(X,Y)~D}\left[{\ell }_{0-1}(Y,h(X))\right]={\mathbf{P}}_{(X,Y)~D}(h(X)\ne Y).$$ (1)

A Bayes consistent algorithm is one which, given enough training examples, learns a classifier whose generalization error approaches the Bayes optimal error:

$${\mathrm{er}}_D^{0-1,*}=\underset{h:\mathcal{X}\to \mathcal{Y}}{\inf }{\mathrm{er}}_D^{0-1}[h].$$ (2)

34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

On the other hand, for a class of models $\mathcal{H}\subset \{h:\mathcal{X}\to \mathcal{Y}\}$, an $\mathcal{H}$-consistent algorithm is one which, given enough training examples, learns a classifier whose generalization error approaches the optimal error in $\mathcal{H}$:

$${\mathrm{er}}_D^{0-1}[\mathcal{H}]=\underset{h\in \mathcal{H}}{\inf }{\mathrm{er}}_D^{0-1}[h].$$ (3)

Since minimizing the discrete 0–1 loss directly is generally computationally hard, a popular approach to multiclass classification is to learn n real-valued scoring functions $f_1,\dots ,f_n:\mathcal{X}\to ℝ$, one for each class, by minimizing a (often convex) surrogate loss, and then given a new test point $\text{x}\in \mathcal{X}$, to predict a class y with highest score f_y(x). Specifically, given a training sample S as above, a surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$, and a scoring function class $\mathcal{F}\subset \left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\right\}$, a $(\psi ,\mathcal{F})$ surrogate risk minimization algorithm finds a vector of n scoring functions $\widehat{\mathbf{f}}:\mathcal{X}\to {ℝ}^n$ by solving

$$\widehat{\mathbf{f}}\in {\mathrm{argmin}}_{\mathbf{f}\in \mathcal{F}}\frac{1}{m}{\sum }_{i=1}^m\psi \left(y_i,\mathbf{f}\left({\mathbf{x}}_i\right)\right),$$ (4)

and then returns a classifier $\widehat{h}:\mathcal{X}\to \mathcal{Y}$ given by

$$\widehat{h}(\mathbf{x})\in {\mathrm{argmax}}_{y\in [n]}{\widehat{f}}_y(\mathbf{x}).$$ (5)

This approach includes several popular multiclass learning algorithms, such as multiclass logistic regression, various forms of multiclass SVMs [15, 4, 6, 5], one-vs-all logistic regression, and one-vs-all SVM; see Table 1 for a summary of the surrogate losses used by some of these algorithms.

Table 1:

Examples of convex surrogate losses used by various multiclass classification algorithms, together with a summary of some previous consistency results (here z₊ = max(0,z)). In this paper, we show that one-vs-all surrogates can in fact achieve $\mathcal{H}$-consistency if minimized over the right scoring function class.

Algorithm	Surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$	Universally calibrated?	Realizable calibrated?	Realizable ${\mathcal{H}}_{\mathrm{lin}}$-consistent?
Multiclass logistic regression	${\psi }_{\mathrm{mlog}}(y,\mathbf{u})=-u_y+\ln \left({\mathrm{\Sigma }}_{y\prime =1}^n\exp \right(u_{y\prime }\left)\right)$	✓	✓	✓
Crammer-Singer multiclass SVM	ψCS(y,u) = maxy'≠y(1 − (uy − uy'))+	×	✓	✓
One-vs-all logistic regression	${\psi }_{\text{OvA,log}}(y,\mathbf{u})=\ln (1+e^{{-u}_y})+{\mathrm{\Sigma }}_{y\prime \ne y}\ln (1+e^{u_{y\prime }})$	✓	✓	× (we give a fix)
One-vs-all SVM	${\psi }_{\text{OvA,hinge}}(y,\mathbf{u})={(1-u_y)}_{+}+{\mathrm{\Sigma }}_{y\prime \ne y}{(1+u_{y\prime })}_{+}$	×	✓	× (we give a fix)

A natural question then is: Under what conditions do such surrogate risk minimization algorithms provide Bayes consistency or, for various classes $\mathcal{H}$ of interest, $\mathcal{H}$-consistency, for the target 0–1 loss?

Surrogate losses and Bayes consistency.

For Bayes consistency, the above question is answered by the notion of calibrated surrogates [2, 17, 16, 14, 13, 11]. Specifically, if a surrogate loss ψ is calibrated w.r.t. the 0–1 loss, then for any universal function class ${\mathcal{F}}_{\text{univ}}$, the $\left(\psi ,{\mathcal{F}}_{\text{univ}}\right)$ surrogate risk minimization algorithm (implemented with suitable regularization) is a Bayes consistent algorithm for ℓ_0–1.¹ Among the surrogate losses shown in Table 1, ψ_mlog and ψ_OvA,log are universally calibrated for ℓ_0–1 (calibrated for all probability distributions), while ψ_CS and ψ_OvA,hinge are calibrated under the so-called ‘dominant-label’ condition (calibrated for distributions in which the conditional distributions p(y|x) assign probability at least $\frac{1}{2}$ to one of the n classes) [16].

Surrogate losses and $\mathcal{H}$-consistency.

For $\mathcal{H}$-consistency, the situation is more complex [3, 7]. In particular, Long and Servedio [7] showed the following results:²

1. Realizable ${\mathcal{H}}_{\text{cls}}$-consistency of Crammer-Singer surrogate for closed-under-scaling models ${\mathcal{H}}_{\text{cls}}$. Let ${\mathcal{F}}_{\text{cls}}\subset \left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\right\}$ be any class of (vector) scoring functions that is closed under scaling, and

   $${\mathcal{H}}_{\text{cls}}=\left\{h:\mathcal{X}\to \mathcal{Y}\mid \exists \text{f}\in {\mathcal{F}}_{\text{cls}}\text{s}.\text{t}.h(\text{x})\in {\mathrm{argmax}}_{y\in [n]}{f}_y(\text{x})\forall \text{x}\right\}.$$

   Long and Servedio [7] showed that if the data distribution D is ${\mathcal{H}}_{\text{cls}}$-realizable (i.e. the data is labeled according to a true model $h^{*}\in {\mathcal{H}}_{\text{cls}}$), then minimizing the Crammer-Singer surrogate ψ_CS over ${\mathcal{F}}_{\text{cls}}$ is ${\mathcal{H}}_{\text{cls}}$-consistent, i.e. the $\left({\psi }_{\text{CS}},{\mathcal{F}}_{\text{cls}}\right)$ surrogate risk minimization algorithm is ${\mathcal{H}}_{\text{cls}}$-consistent. This was viewed as surprising in light of the fact that ψ_CS is not (universally) calibrated for ℓ_0–1.

2. Lack of realizable ${\mathcal{H}}_{\mathrm{lin}}$-consistency of one-vs-all logistic surrogate for linear models ${\mathcal{H}}_{\mathrm{lin}}$. Let ${\mathcal{F}}_{\mathrm{lin}}$ be the class of linear (vector) scoring functions and ${\mathcal{H}}_{\mathrm{lin}}$ the class of linear multiclass classification models:

   $${\mathcal{F}}_{\mathrm{lin}}=\left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\mid \exists {\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d\text{s}.\text{t}.f_y(\mathbf{x})={\mathbf{w}}_y^{\top }\mathbf{x}\forall \mathbf{x}\right\}$$ (6)

   $${\mathcal{H}}_{\mathrm{lin}}=\left\{h:\mathcal{X}\to \mathcal{Y}\mid \exists {\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d\text{s}.\text{t}.h(\text{x})\in {\mathrm{argmax}}_{y\in [n]}{\mathbf{w}}_y^{\top }\mathbf{x}\forall \mathbf{x}\right\}.$$ (7)

Long and Servedio [7] showed that even if the data distribution D is ${\mathcal{H}}_{\mathrm{lin}}$-realizable (i.e. the data is labeled according to a true linear model $h^{*}\in {\mathcal{H}}_{\text{lin}}$), minimizing the one-vs-all logistic surrogate ψ_OvA,log over ${\mathcal{F}}_{\mathrm{lin}}$ fails to give an ${\mathcal{H}}_{\mathrm{lin}}$-consistent algorithm, i.e. the (ψ_OvA,log, ${\mathcal{F}}_{\mathrm{lin}}$) surrogate risk minimization algorithm is not ${\mathcal{H}}_{\mathrm{lin}}$-consistent, even though ψ_OvA,log is universally calibrated for ℓ_0–1.

Our contributions.

As discussed by Long and Servedio [7] and summarized above, it seems peculiar that the Crammer-Singer surrogate ψ_CS, which is not universally calibrated for ℓ_0–1, provides realizable ${\mathcal{H}}_{\mathrm{lin}}$-consistency (and more generally, realizable ${\mathcal{H}}_{\mathrm{cls}}$-consistency for closed-under-scaling models ${\mathcal{H}}_{\mathrm{cls}}$), while the one-vs-all logistic surrogate, ψ_OvA,log, which is universally calibrated for ℓ_0–1, fails to provide realizable ${\mathcal{H}}_{\mathrm{lin}}$-consistency. In this paper, we investigate this apparent conundrum.

First, regarding result (1) of Long and Servedio [7] above, we note that any realizable distribution D (i.e. a distribution that labels data points x according to a deterministic model y = h(x)) trivially satisfies the dominant-label condition (for each x, one class y has conditional probability $p(y\mid \mathbf{x})\ge \frac{1}{2}$), and therefore the Crammer-Singer surrogate ψ_CS is in fact calibrated for any such distribution. Therefore, in the realizable setting studied by Long and Servedio [7], the surrogate ψ_CS is in fact calibrated for ℓ_0–1 (the paper emphasizes that ψ_CS is not calibrated/consistent for ℓ_0–1, implicitly referring to universal calibration, and misses the fact that it is indeed calibrated for the setting studied). So, while the result (1) is still interesting and non-trivial, it should be kept in mind that under the realizable setting studied in [7], all the surrogates studied by the authors are in fact calibrated for ℓ_0–1.

Second, and more importantly, we look into result (2) of Long and Servedio [7] above. We know that minimizing the one-vs-all logistic surrogate ψ_OvA,log over a universal scoring function class ${\mathcal{F}}_{\text{univ}}$ gives Bayes consistency for all distributions D. Therefore, for ${\mathcal{H}}_{\mathrm{lin}}$-realizable distributions D, where ${\mathrm{er}}_D^{0-1,*}={\mathrm{er}}_D^{0-1}\left[{\mathcal{H}}_{\text{lin}}\right]$ and therefore Bayes consistency is equivalent to ${\mathcal{H}}_{\mathrm{lin}}$-consistency, we have that minimizing the ψ_OvA,log surrogate over such a class ${\mathcal{F}}_{\text{univ}}$ gives an ${\mathcal{H}}_{\mathrm{lin}}$-consistent algorithm. So why does minimizing the same surrogate over the class ${\mathcal{F}}_{\mathrm{lin}}$ of linear scoring functions fail in this regard? On closer inspection, we find that an important part of the answer lies in the form of the decision boundaries induced by a linear (or more generally, affine) multiclass classification model. As an example, Figure 1 shows an affine 4-class model in a 2-dimensional instance space; specifically, the figure shows the 4 affine scoring functions for the 4 classes, and the corresponding decision regions. As can be seen, the one-vs-all boundaries induced by such a model are not linear! Indeed, in general, each class is described by a convex polytope, and is separated from the rest of the classes by a piecewise linear decision boundary (where the boundaries for different classes include shared pieces). Therefore, when the one-vs-all classifier is forced to separate each class from the rest using a linear decision boundary, it can end up learning a suboptimal separator.

Figure 1:

Example in d = 2 dimensions with n = 4 classes. Top row: True linear 4-class classifier $h^{*}\in {\mathcal{H}}_{\text{lin}}$. The first 4 plots show contours of the 4 linear scoring functions $f_1^{*},\dots ,f_4^{*}:\mathcal{X}\to ℝ$ (darker shades represent higher values); the 5th plot shows the regions corresponding to the classifier $h^{*}(x)\in {\mathrm{argmax}}_{y\in [n]}{f}_y^{*}(x)$. As can be seen, each class is described by a convex polytope, and is separated from the rest by a piecewise linear decision boundary. Middle row: Contours of scoring functions and decision regions learned from training data labeled according to h^∗ by minimizing the one-vs-all logistic surrogate ψ_OvA,log over the linear scoring function class ${\mathcal{F}}_{\text{lin}}$. The generalization accuracy is 0.886. Bottom row: Contours of scoring functions and decision regions learned from the same training data by minimizing the same one-vs-all logistic surrogate ψ_OvA,log over the ‘shared’ piecewise linear scoring function class ${\mathcal{F}}_{\text{spwlin}}$. The decision regions are closer to the true model, and the generalization accuracy is 0.986. Since the piecewise linear functions have shared pieces, the number of parameters to be learned is the same as in the linear case; moreover, the resulting model can also be transformed to a linear model if desired (see Theorem 3 and Corollary 4). See Section 5 for details of the experimental setup.

In the rest of the paper, we use the above insight to design a special class of ‘shared’ piecewise linear scoring functions ${\mathcal{F}}_{\text{spwlin}}$ such that minimizing the one-vs-all logistic surrogate ψ_OvA,log over this class yields an ${\mathcal{H}}_{\mathrm{lin}}$-consistent algorithm. We will see that ${\mathcal{F}}_{\text{spwlin}}$ is characterized by the same number of parameters as ${\mathcal{F}}_{\mathrm{lin}}$; in fact, ${\mathcal{F}}_{\text{spwlin}}$ will also be parametrized by n weight vectors ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d$.³ In order to minimize ψ_OvA,log over this scoring function class, we will make use of an adaptation of the min-pooling idea from neural network training. The same idea can be applied to other one-vs-all surrogates as well; in our experiments, we consider both ψ_OvA,log and the one-vs-all SVM surrogate ψ_OvA,hinge, and find that in both cases, while minimizing these surrogates over the class of linear scoring functions ${\mathcal{F}}_{\mathrm{lin}}$ fails to provide ${\mathcal{H}}_{\mathrm{lin}}$-consistency, minimizing them over the nonlinear scoring function class ${\mathcal{F}}_{\text{spwlin}}$ does indeed provide ${\mathcal{H}}_{\mathrm{lin}}$-consistency.

An additional interesting aspect of the scoring function class ${\mathcal{F}}_{\text{spwlin}}$ is that, while the individual scoring functions in it are nonlinear (specifically, piecewise linear), the classification models resulting from taking the highest-scoring class according to these scoring functions can also be expressed as linear models. Therefore, having learned a classifier by minimizing a one-vs-all surrogate over this nonlinear scoring function class, one can then convert the learned model to a linear model in ${\mathcal{H}}_{\mathrm{lin}}$.

We believe our study can pave the way for a more thorough understanding of the role of surrogate losses in $\mathcal{H}$-consistency. In particular, our results suggest that, when studying $\mathcal{H}$-consistency, one needs to carefully take into account the interplay between surrogate losses and the scoring function class over which they are minimized, and that this can lead to unexpected improvements to learning algorithms used in practice.

Organization.

We start by giving various formal definitions in Section 2. We then describe the class of ‘shared’ piecewise linear scoring functions ${\mathcal{F}}_{\text{spwlin}}$ and give our associated consistency result in Section 3. We discuss how to minimize one-vs-all surrogates over this scoring function class in practice in Section 4, and describe our numerical experiments in Section 5. Section 6 concludes with a brief summary. Additional details/proofs are provided in the supplementary material.

2. Formal Definitions: Consistency, Calibration, Realizability

Consistency.

We start with formal definitions of Bayes consistency and $\mathcal{H}$-consistency:

Definition 1 (Bayes consistency).

We say a multiclass learning algorithm that maps training samples $S\in {\cup }_{m=1}^{\infty }{(\mathcal{X}\times \mathcal{Y})}^m$ to multiclass models $\widehat{h}:\mathcal{X}\to \mathcal{Y}$ is Bayes consistent (w.r.t. ℓ_0–1) for a distribution D if for all 𝜖 > 0,

$$\underset{m\to \infty }{\lim }{\mathbf{P}}_{S~D^m}\left({\mathrm{er}}_D^{0-1}[\widehat{h}]-{\mathrm{er}}_D^{0-1,*}>ϵ\right)=0.$$

If an algorithm is Bayes consistent for all distributions D, we say it is universally Bayes consistent.

Definition 2 ( $\mathcal{H}$-consistency).

Let $\mathcal{H}\subset \{h:\mathcal{X}\to \mathcal{Y}\}$. We say a multiclass learning algorithm that maps training samples $S\in {\cup }_{m=1}^{\infty }{(\mathcal{X}\times \mathcal{Y})}^m$ to multiclass models $\widehat{h}:\mathcal{X}\to \mathcal{Y}$ is $\mathcal{H}$-consistent (w.r.t. ℓ_0–1) for a distribution D if for all 𝜖 > 0,

$$\underset{m\to \infty }{\lim }{\mathbf{P}}_{S~D^m}\left({\mathrm{er}}_D^{0-1}[\widehat{h}]-{\mathrm{er}}_D^{0-1}[\mathcal{H}]>ϵ\right)=0.$$

Note that we do not require the algorithm to produce a model in $\mathcal{H}$; we only require that as m→∞, the performance of the model it learns approaches that of the best model in $\mathcal{H}$. If an algorithm is $\mathcal{H}$-consistent for all distributions D, we say it is universally $\mathcal{H}$-consistent.

Calibration.

Next, we give the standard definition of calibration of a surrogate loss that is useful for studying Bayes consistency of surrogate risk minimization algorithms, followed by a definition of calibration w.r.t. $\mathcal{H}$ that is useful for studying $\mathcal{H}$-consistency of such algorithms. To give these definitions, for a surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$, we need the following notions of ψ-generalization error, Bayes optimal ψ-error, and optimal ψ-error in a scoring function class $\mathcal{F}\subset \left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\right\}$:

$${\mathrm{er}}_D^{\psi }[\text{f}]={\mathbf{E}}_{(X,Y)~D}[\psi (Y,\mathbf{f}(X))];{\mathrm{er}}_D^{\psi ,*}=\underset{\text{f}:\mathcal{X}\to {ℝ}^n}{\inf }{\mathrm{er}}_D^{\psi }[\mathbf{f}];{\mathrm{er}}_D^{\psi }[\mathcal{F}]=\underset{\mathbf{f}\in \mathcal{F}}{\inf }{\mathrm{er}}_D^{\psi }[\mathbf{f}].$$ (8)

Definition 3 (Calibration (standard definition)).

We say a surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$ is calibrated w.r.t. ℓ_0–1 for a distribution D if there exists a strictly increasing function $g:{ℝ}_{+}\to {ℝ}_{+}$ that is continuous at 0 with g(0) = 0 such that for all $\text{f}:\mathcal{X}\to {ℝ}^n$,

$${\mathrm{er}}_D^{0-1}[\underset{h}{\underbrace{\mathrm{argmax}\circ \mathbf{f}}}]-{\mathrm{er}}_D^{0-1,*}\le g\left({\mathrm{er}}_D^{\psi }[\mathbf{f}]-{\mathrm{er}}_D^{\psi ,*}\right),$$

where h ≡ argmax ∘ f denotes a classifier that satisfies $h(\text{x})\in {\mathrm{argmax}}_{y\in [n]}{f}_y(\text{x})$. If ψ is calibrated w.r.t. ℓ_0–1 for all distributions D, we say ψ is universally calibrated w.r.t. ℓ_0–1.

Definition 4 (Calibration w.r.t. $\mathcal{H}$).

For a class of multiclass models $\mathcal{H}\subset \{h:\mathcal{X}\to \mathcal{Y}\}$, a surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$, and a scoring function class $\mathcal{F}\subset \left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\right\}$, we say $(\psi ,\mathcal{F})$ is calibrated w.r.t. $\left({\ell }_{0-1},\mathcal{H}\right)$ for a distribution D if there exists a strictly increasing function $g:{ℝ}_{+}\to {ℝ}_{+}$ that is continuous at 0 with g(0) = 0 such that for all $f\in \mathcal{F}$,

$${\mathrm{er}}_D^{0-1}[\underset{h}{\underbrace{\mathrm{argmax}\circ \mathbf{f}}}]-{\mathrm{er}}_D^{0-1}[\mathcal{H}]\le g\left({\mathrm{er}}_D^{\psi }[\mathbf{f}]-{\mathrm{er}}_D^{\psi }[\mathcal{F}]\right),$$

where h ≡ argmax ∘ f denotes a classifier that satisfies $h(\text{x})\in {\mathrm{argmax}}_{y\in [n]}{f}_y(\text{x})$. If $(\psi ,\mathcal{F})$ is calibrated w.r.t. $\left({\ell }_{0-1},\mathcal{H}\right)$ for all distributions D, we say $(\psi ,\mathcal{F})$ is universally calibrated w.r.t. $\left({\ell }_{0-1},\mathcal{H}\right)$.

Realizability and realizable calibration/consistency.

Finally, we give formal definitions of realizable and $\mathcal{H}$-realizable distributions, realizable calibration, and Long and Servedio’s definition of realizable ${\mathcal{H}}_{\mathcal{F}}$-consistency.

Definition 5 (Realizability and $\mathcal{H}$-realizability).

We say a distribution D over $\mathcal{X}\times \mathcal{Y}$ is realizable if (almost surely) it labels points according to a deterministic model, i.e. if $\exists h:\mathcal{X}\to \mathcal{Y}$ such that $P_{(X,Y)~D}(h(X)=Y)=1$. For a class $\mathcal{H}\subset \{h:\mathcal{X}\to \mathcal{Y}\}$, we say a distribution D over $\mathcal{X}\times \mathcal{Y}$ is $\mathcal{H}$-realizable if (almost surely) it labels points according to a deterministic model in $\mathcal{H}$, i.e. if $\exists h\in \mathcal{H}$ such that $P_{(X,Y)~D}(h(X)=Y)=1$.

Definition 6 (Realizable calibration).

We say a surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$ is realizable calibrated (w.r.t. ℓ_0–1) if it is calibrated (w.r.t. ℓ_0–1) for all realizable distributions.⁴

Definition 7 (Long and Servedio’s definition of realizable ${\mathcal{H}}_{\mathcal{F}}$-consistency [7]).

Let $\mathcal{F}\subset \left\{f:\mathcal{X}\to {ℝ}^n\right\}$, and let ${\mathcal{H}}_{\mathcal{F}}=\{h:\mathcal{X}\to \mathcal{Y}\mid \exists \mathbf{f}\in \mathcal{F}\text{s}.\text{t}.h(\mathbf{x})\in {\mathrm{argmax}}_y{f}_y(\mathbf{x})\forall \mathbf{x}\}$. A surrogate loss $\psi :\mathcal{Y}\times {ℝ}^n\to {ℝ}_{+}$ is realizable ${\mathcal{H}}_{\mathcal{F}}$-consistent if $(\psi ,\mathcal{F})$ is calibrated w.r.t. $\left({\ell }_{0-1},{\mathcal{H}}_{\mathcal{F}}\right)$ for all ${\mathcal{H}}_{\mathcal{F}}$-realizable distributions.^5,6

3. Minimizing One-vs-All Surrogates over a Class ${\mathcal{F}}_{\mathrm{spwlin}}$ of ‘Shared’ Piecewise Linear Scoring Functions is ${\mathcal{H}}_{\mathrm{lin}}$-Consistent

As discussed in Section 1, even though the one-vs-all logistic surrogate ψ_OvA,log is universally calibrated for ℓ_0–1, Long and Servedio [7] showed that the $\left({\psi }_{\text{OvA},\log },{\mathcal{F}}_{\text{lin}}\right)$ surrogate risk minimization algorithm, which minimizes ψ_OvA,log over the class of linear scoring functions ${\mathcal{F}}_{\text{lin}}$, is not ${\mathcal{H}}_{\text{lin}}$-consistent even when the data distribution D is ${\mathcal{H}}_{\text{lin}}$-realizable. In this section, we remedy this situation by showing how to minimize the same surrogate loss ψ_OvA,log (as well as other one-vs-all surrogate losses) over a different, nonlinear scoring function class ${\mathcal{F}}_{\text{spwlin}}$ such that the resulting algorithm is ${\mathcal{H}}_{\text{lin}}$-consistent for all ${\mathcal{H}}_{\text{lin}}$-realizable distributions D.

Linear models.

For the remainder of the paper, we will re-define the classes of linear scoring functions and linear classification models to allow for the inclusion of bias/offset terms:

$${\mathcal{F}}_{\text{lin}}=\left\{\mathbf{f}:\mathcal{X}\to {ℝ}^n\mid \exists {\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d,b_1\dots ,b_n\in ℝ\text{s}.\text{t}.f_y(\mathbf{x})={\mathbf{w}}_y^{\top }\mathbf{x}+b_y\forall \mathbf{x}\right\}$$ (9)

$${\mathcal{H}}_{\text{lin}}=\left\{h:\mathcal{X}\to \mathcal{Y}\mid \exists {\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d,b_1\dots ,b_n\in ℝ\text{s}.\text{t}.h(\mathbf{x})\in {\mathrm{argmax}}_{y\in [n]}{\mathbf{w}}_y^{\top }\mathbf{x}+b_y\forall \mathbf{x}\right\}.$$ (10)

Our conclusions will apply both in this more general setting, and in the special case where b_y = 0 ∀y.

‘Shared’ piecewise linear scoring functions.

To motivate the scoring function class we will construct, consider again the example in Figure 1. As this example makes clear, under a linear classification model in ${\mathcal{H}}_{\text{lin}}$ defined by weight vectors ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d$ and bias terms $b_1\dots ,b_n\in ℝ$, the decision region corresponding exclusively to class y ∈ [n] is the (open) convex polytope given by

$${\mathcal{X}}_y=\left\{\mathbf{x}\in \mathcal{X}\mid {\mathbf{w}}_y^{\top }\mathbf{x}+b_y>{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\forall y^{\prime }\ne y\right\}$$ (11)

$$=\left\{\mathbf{x}\in \mathcal{X}\mid {\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)>0\forall y^{\prime }\ne y\right\}$$ (12)

$$=\left\{\text{x}\in \mathcal{X}\mid {\min }_{y^{\prime }\ne y}\left\{{\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)\right\}>0\right\}.$$ (13)

We use this observation to construct the following special class of ‘shared’ piecewise linear scoring functions:

$$\begin{gathered} {\mathcal{F}}_{\text{spwlin}}=\{\text{f}:\mathcal{X}\to {ℝ}^n\mid \exists {\text{w}}_1,\dots ,{\text{w}}_n\in {ℝ}^d,b_1\dots ,b_n\in ℝ\text{s}.\text{t}. \\ {f}_y(\text{x})={\min }_{y^{\prime }\ne y}\left\{{\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)\right\}\forall \mathbf{x}\}. \end{gathered}$$ (14)

Clearly, this class is parametrized by the same number of parameters as ${\mathcal{F}}_{\text{lin}}$. The reason that the class ${\mathcal{F}}_{\text{spwlin}}$ is useful is that the scoring functions in this class will allow for learning precisely the form of one-vs-all decision boundaries that are induced by linear multiclass models. In particular, we have the following result:

Lemma 1

(Scoring functions in ${\mathcal{F}}_{\text{spwlin}}$ capture correct one-vs-all decision boundaries for linear multiclass models). Let $\mathbf{f}\in {\mathcal{F}}_{\text{spwlin}}$ be parametrized by ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d$, $b_1,\dots ,b_n\in ℝ$. Then

$$f_y(\text{x})\ge 0\iff y\in {\mathrm{argmax}}_{y^{\prime }\in [n]}{\mathbf{w}}_{y^{\prime }}^{\top }\text{x}+b_{y^{\prime }}.$$ (15)

Since one-vs-all surrogates effectively learn scoring functions that aim to separate points x with label y from points with other labels according to whether f_y(x) ≥ 0, the above result implies that minimizing such surrogates over the class ${\mathcal{F}}_{\text{spwlin}}$ should allow learning precisely the form of one-vs-all separation boundaries induced by linear multiclass models. Formally, we have the following ${\mathcal{H}}_{\text{lin}}$-consistency result:

Theorem 2

( ${\mathcal{H}}_{\text{lin}}$-consistency of $\left({\psi }_{\text{OvA,log}},{\mathcal{F}}_{\text{spwlin}}\right)$ surrogate risk minimization algorithm). The pair $\left({\psi }_{\text{OvA,log}},{\mathcal{F}}_{\text{spwlin}}\right)$ is calibrated w.r.t. $\left({\ell }_{0-1},{\mathcal{H}}_{\text{lin}}\right)$ for all ${\mathcal{H}}_{\text{lin}}$-realizable distributions.

Remark 1

(Generalization to other one-vs-all surrogates). The above ${\mathcal{H}}_{\text{lin}}$-consistency result can be generalized to other one-vs-all surrogates, such as the one-vs-all hinge surrogate ψ_OvA,hinge.

Remark 2

(Loss of ‘independence’ of one-vs-all binary classifiers). Since the n components of the (vector) scoring functions in ${\mathcal{F}}_{\text{spwlin}}$ share parameters, they can no longer be learned independently by training separate binary classifiers in parallel; while minimizing a one-vs-all surrogate over ${\mathcal{F}}_{\text{spwlin}}$ still amounts to learning binary separators for each of the classes versus the rest, these separators must be learned together in an “all-in-one” multiclass learning algorithm.

Remark 3

(Non-convexity of resulting optimization problems). Although the one-vs-all surrogates ψ_OvA,log and ψ_OvA,hinge are convex, minimizing these surrogates over the function class ${\mathcal{F}}_{\text{spwlin}}$ results in non-convex optimization problems. In order to solve these optimization problems, our implementation makes use of an adaptation of the min-pooling idea from neural network training (see Section 4). Additional details regarding the behavior of this approach in our experiments are discussed in Section 5.

We also have the following result, which shows that the classification models induced by the nonlinear (vector) scoring functions in ${\mathcal{F}}_{\text{spwlin}}$ are in fact equivalent to those in the class of linear classification models ${\mathcal{H}}_{\text{lin}}$:

Theorem 3

(Scoring functions in ${\mathcal{F}}_{\text{spwlin}}$ induce linear multiclass classifiers). Let ${\mathcal{H}}_{\text{spwlin}}$ be the class of multiclass classifiers induced by ${\mathcal{F}}_{\text{spwlin}}$:

$${\mathcal{H}}_{\text{spwlin}}=\left\{h:\mathcal{X}\to \mathcal{Y}\mid \exists \mathbf{f}\in {\mathcal{F}}_{\text{spwlin}}s.t.h(\mathbf{x})\in {\mathrm{argmax}}_{y\in [n]}{f}_y(\mathbf{x})\right\}.$$ (16)

Then ${\mathcal{H}}_{\text{spwlin}}={\mathcal{H}}_{\text{lin}}$.

Indeed, the following corollary shows that once we have learned a nonlinear (vector) scoring function in ${\mathcal{F}}_{\text{spwlin}}$, we can easily transform it into a linear classification model in ${\mathcal{H}}_{\text{lin}}$:

Corollary 4

(Converting a nonlinear scoring function in ${\mathcal{F}}_{\text{spwlin}}$ to a linear classification model in ${\mathcal{H}}_{\text{lin}}$). Let $\mathbf{f}\in {\mathcal{F}}_{\text{spwlin}}$ be parametrized by ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_n\in {ℝ}^d$, $b_1,\dots ,b_n\in ℝ$. Then for all $x\in \mathcal{X}$,

$${\mathrm{argmax}}_{y\in [n]}{f}_y(\mathbf{x})={\mathrm{argmax}}_{y\in [n]}{\mathbf{w}}_y^{\top }\mathbf{x}+b_y.$$ (17)

Margin interpretation of ${\mathcal{F}}_{spwlin}$.

We note that the scoring functions in ${\mathcal{F}}_{\text{spwlin}}$ can also be viewed as computing a multiclass ‘margin’ vector over the underlying linear functions defining the shared piecewise linear scores. Specifically, recall that a (vector) scoring function $\mathbf{f}\in {\mathcal{F}}_{\text{spwlin}}$ has the form

$$f_y(\text{x})=\underset{y^{\prime }\ne y}{\min }\left\{{\left({\text{w}}_y-{\text{w}}_{y^{\prime }}\right)}^{\top }\text{x}+\left(b_y-b_{y^{\prime }}\right)\right\}=\left({\text{w}}_y^{\top }\text{x}+b_y\right)-\underset{y^{\prime }\ne y}{\min }\left\{{\mathbf{w}}_{y^{\prime }}^{\top }\text{x}+b_{y^{\prime }}\right\}$$ (18)

for some ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^n$This suggests that for each y, the score f_y(x) effectively computes the ‘margin’ of separation between $\left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)$ and ${\max }_{y^{\prime }\ne y}\left\{{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right\}$; if this margin is non-negative, then $y\in {\mathrm{argmax}}_{y^{\prime }\in [n]}{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}$, and if it is negative, then $y\notin {\mathrm{argmax}}_{y^{\prime }\in [n]}{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}$.

Generalization to other multiclass models $\mathcal{H}$.

The above construction can be generalized beyond ${\mathcal{H}}_{\mathrm{lin}}$ to other multiclass models ${\mathcal{H}}_{\mathcal{G}}$ defined in terms of a class of real-valued scoring functions $\mathcal{G}\subset \{g:\mathcal{X}\to ℝ\}$. Specifically, for any such class $\mathcal{G}$, let

$${\mathcal{H}}_{\mathcal{G}}=\left\{h:\mathcal{X}\to \mathcal{Y}\mid \exists g_1,\dots ,g_n\in \mathcal{G}\text{s}.\text{t}.h(\mathbf{x})\in {\mathrm{argmax}}_{y\in [n]}{g}_y(\mathbf{x})\forall \mathbf{x}\right\}.$$ (19)

(Thus ${\mathcal{H}}_{\mathrm{lin}}$ is a special case with ${\mathcal{G}}_{\text{lin}}=\left\{g:\mathcal{X}\to ℝ\mid \exists \mathbf{w}\in {ℝ}^d,b\in ℝ\text{s}.\text{t}.g(\mathbf{x})={\mathbf{w}}^{\top }\mathbf{x}+b\forall \mathbf{x}\right\}$.) Define the class of ‘shared’ piecewise-difference-of- $\mathcal{G}$ scoring functions ${\mathcal{F}}_{\text{spwdiff}\mathcal{G}}$ as follows:

$${\mathcal{F}}_{\text{spwdiff}\mathcal{G}}\text{=}\left\{\mathbf{f}:\mathcal{X}\to {ℝ}^{\text{n}}\mid \exists {\text{g}}_{\text{1}},\dots ,{\text{g}}_{\text{n}}\in \mathcal{G}\text{s}.\text{t}{.\text{f}}_{\text{y}}(\mathbf{x})\text{=}\underset{y^{\prime }\ne y}{\min }\left\{{\text{g}}_{\text{y}}(\mathbf{x}){\text{-g}}_{{\text{y}}^{\prime }}(\mathbf{x})\right\}\forall \mathbf{x}\right\}.$$ (20)

Then similarly to the linear case, it can be shown that minimizing any of the one-vs-all surrogates ψ_OvA,log or ψ_OvA,hinge over ${\mathcal{F}}_{\text{spwdiff}\mathcal{G}}$ is ${\mathcal{H}}_{\mathcal{G}}$-consistent for all ${\mathcal{H}}_{\mathcal{G}}$-realizable distributions.

4. Implementation of One-vs-All Surrogate Risk Minimization over ${\mathcal{F}}_{\mathrm{spwlin}}$

In order to implement surrogate risk minimization over the scoring function class ${\mathcal{F}}_{\text{spwlin}}$, we make use of an adaptation of the min-pooling idea from neural network training. Figure 2 shows a summary of the architecture we use to implement scoring functions f in ${\mathcal{F}}_{\text{spwlin}}$.

Figure 2:

Neural network-like architecture implementing scoring functions in ${\mathcal{F}}_{\text{spwlin}}$. To find parameters ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^n$ minimizing a surrogate loss ψ on the training data, we use a backpropagation-like procedure on this architecture.

Specifically, given an input point $\mathbf{x}\in {ℝ}^d$, the first layer computes the n linear functions

$$g_y(\text{x})={\mathbf{w}}_y^{\top }\text{x}+b_y,y\in [n].$$

The second layer then computes the n scoring function components f_y(x) in terms of minima of the relevant functions from the first layer (see Eq. (14)):

$${\mu }_y(\mathbf{g})=\underset{y^{\prime }\ne y}{\min }\left\{g_y-g_{y^{\prime }}\right\},y\in [n].$$

To fit the parameters ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^n$ to training data, we then use a backpropagation-like procedure to minimize the surrogate loss of interest. Any existing neural network training library can be easily modified to perform this minimization; in our experiments, we implemented this approach using PyTorch [10].

5. Experiments

We conducted two sets of experiments. In the first set, we generated synthetic data from a true linear model (i.e. a known ${\mathcal{H}}_{\text{lin}}$-realizable distribution) and tested the ${\mathcal{H}}_{\text{lin}}$-consistency of minimizing one-vs-all surrogates over ${\mathcal{F}}_{\text{spwlin}}$. In the second set, we implemented the approach on various real benchmark data sets to test its practical behavior. In all cases, we implemented a total of 6 multiclass algorithms: all 4 algorithms shown in Table 1 with surrogate risk minimized over linear scoring functions ${\mathcal{F}}_{\text{lin}}$, and the two one-vs-all algorithms with surrogate risk minimized over ${\mathcal{F}}_{\text{spwlin}}$. All algorithms were implemented in PyTorch and used the AdamW optimizer [8].^7,8

5.1. Synthetic Data: Consistency Behavior on Linear Models

We generated two synthetic data sets. The first data set had d = 2 features and n = 4 classes. A true model $h^{*}\in {\mathcal{H}}_{\text{lin}}$ was created by choosing ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^4$ as follows: elements of ${\mathbf{w}}_y\in {ℝ}^2$ were drawn i.i.d. from $\mathcal{N}(0,1)$ and subsequently scaled so that ||w_y||₂ = 1 ∀y; bias terms b₁, …, b₄ were set to 0.2, 0.1, −0.1, −0.2 (decision regions of the resulting model h^∗ are shown in Figure 1). Instances x were then drawn uniformly at random from a disk of radius 0.5 centered at (0.3,−0.1), and labeled according to h^∗. We ran all 6 algorithms (using AdamW with zero weight decay factor) on increasingly large training samples (up to 30,000 data points) generated in this manner, and measured the generalization accuracy on a large test set of 10,000 data points generated in the same manner. The results are shown in Figure 3 (left); an illustration of some of the models learned from 10,000 data points is also shown in Figure 1.

Figure 3:

Convergence behavior of various multiclass surrogate risk minimization algorithms on synthetic data generated from a true linear model in ${\mathcal{H}}_{\text{lin}}$. Left: d = 2, n = 4. Right: d = 100, n = 10. In both cases, the ψ_OvA,log and ψ_OvA,hinge surrogates fail to converge to the optimal performance in ${\mathcal{H}}_{\text{lin}}$ when minimized over standard linear scoring functions ${\mathcal{F}}_{\text{lin}}$, but successfully do so when minimized over the class ${\mathcal{F}}_{\text{spwlin}}$. (In the right plot, the curves for all four ${\mathcal{H}}_{\text{lin}}$-consistent algorithms overlap.) See Section 5.1 for details.

The second data set had d = 100 features and n = 10 classes. A true model $h^{*}\in {\mathcal{H}}_{\text{lin}}$ was created in the same manner as above, except that in this case we set b_y = 0 ∀y ∈ [100]. Instances x were drawn uniformly at random from X = [−1, 1]¹⁰⁰, and labeled according to h^∗. We ran all 6 algorithms on increasingly large training samples (up to 40,000 data points) and measured accuracy on a large test set of 10,000 data points. The results are shown in Figure 3 (right).

In both cases, the one-vs-all surrogates fail to give ${\mathcal{H}}_{\text{lin}}$-consistency when minimized over linear scoring functions ${\mathcal{F}}_{\text{lin}}$, but successfully do so when minimized over the scoring function class ${\mathcal{F}}_{\text{spwlin}}$.

5.2. Real Data: Practical Behavior

We evaluated the performance of all 6 algorithms on various benchmark multiclass classification data sets drawn from the UCI repository and the LIBSVM data repository.⁹ Details of the data sets are provided in the supplementary material; the number of features d ranges from 16 to 3072, and the number of classes n ranges from 7 to 26. Several of the data sets come with prescribed train/validation/test splits; for the others, we randomly chose a 3:1:1 split. For all algorithms, we used AdamW with a weight decay factor λ; the factor λ was chosen from {10⁻³, …, 10²} to maximize 0–1 accuracy on the validation set.

The results are shown in Table 2. For each data set, the best-performing algorithms within the group of logistic surrogates and within the group of hinge surrogates are shown in bold font; the best overall is enclosed in asterisks. For hinge surrogates, consistent with previous results [5], we find ψ_OvA,hinge, when minimized over ${\mathcal{F}}_{\text{lin}}$, does slightly poorer than ψ_CS, but minimizing it over ${\mathcal{F}}_{\text{spwlin}}$ brings it in line with (and even slightly exceeds) ψ_CS. For logistic surrogates, the results are more mixed, although $\left({\psi }_{\text{OvA,log}},{\mathcal{F}}_{\text{spwlin}}\right)$ frequently outperforms $\left({\psi }_{\text{OvA,log}},{\mathcal{F}}_{\text{lin}}\right)$. Overall, despite the good performance, we do not necessarily advocate minimizing one-vs-all surrogates over ${\mathcal{F}}_{\text{spwlin}}$ as a practical strategy, as training is 2–3 times slower than for ψ_CS or ψ_mlog, which generally give comparable results. Our primary interest is in the ${\mathcal{H}}_{\text{lin}}$-consistency of this scheme under ${\mathcal{H}}_{\text{lin}}$-realizable data distributions; the main purpose of the experiments on real data was to serve as a sanity check and ensure that this does not come at a huge price in terms of practical applicability of the resulting algorithms.

Table 2:

Results (in terms of test accuracy) on various real multiclass data sets. See Section 5.2 for details.

Data set	ψmlog	ψOvA,log	ψOvA,log	ψCS	ψOvA,hinge	ψOvA,hinge
	${\mathcal{F}}_{\mathrm{lin}}$	${\mathcal{F}}_{\mathrm{lin}}$	${\mathcal{F}}_{\mathrm{spwlin}}$	${\mathcal{F}}_{\mathrm{lin}}$	${\mathcal{F}}_{\mathrm{lin}}$	${\mathcal{F}}_{\mathrm{spwlin}}$
Covertype (50K)	0.6606	0.6943	0.6607	0.7186	0.7069	*0.7193*
Digits	0.8985	0.8696	0.8982	0.9025	0.8819	*0.9042*
USPS	*0.9153*	0.9138	0.9148	0.9128	0.9063	0.9148
MNIST (70K)	0.9270	0.9200	0.9271	0.9307	0.9216	*0.9317*
CIFAR10	0.4000	*0.4066*	0.3763	0.3831	0.3686	0.4006
Sensorless	*0.8266*	0.6539	0.7918	0.7703	0.5381	0.7791
Letter	0.7644	0.7126	0.7662	0.7738	0.6058	*0.7804*

6. Conclusion

Our study shows that when studying $\mathcal{H}$-consistency of surrogate risk minimization algorithms, the interplay between the surrogate loss and scoring function class can play an important role. In particular, for ψ_OvA,log and ψ_OvA,hinge, we found that minimization over a suitable function class ${\mathcal{F}}_{\text{spwlin}}$ gives ${\mathcal{H}}_{\text{lin}}$-consistency where standard minimization over linear functions ${\mathcal{F}}_{\text{lin}}$ fails to do so.

Broader Impact.

The primary goal of this paper is to better understand the statistical consistency properties of surrogate risk minimization algorithms in machine learning. The insights and results of the paper will benefit readers who wish to be aware of these properties when designing or selecting learning algorithms.

We do not expect this research to put anyone at a disadvantage. Nevertheless, issues related to data bias and fairness can potentially affect any algorithm that learns models from data [9], and users should keep this in mind when applying the ideas discussed here to domains where such issues may be important. In the future, it may also be of interest to consider incorporating fairness constraints in the types of algorithms discussed here.

Appendix

A. Proof of Lemma 1

Proof.

This essentially follows from the definition of ${\mathcal{F}}_{\text{spwlin}}$. In particular, we have:

$$\begin{gathered} f_y(\mathbf{x})\ge 0\iff \underset{y^{\prime }\ne y}{\min }\left\{{\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)\right\}\ge 0 \\ \iff \underset{y^{\prime }\ne y}{\min }\left\{\left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)-\left({\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right)\right\}\ge 0 \\ \iff \left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)\ge \left({\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right)\forall y^{\prime }\ne y \\ \iff y\in {\mathrm{argmax}}_{y^{\prime }\in [n]}{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}. \end{gathered}$$

B. Proof of Theorem 2

Proof.

Let D be a ${\mathcal{H}}_{\text{lin}}$-realizable distribution. Then $\exists h^{*}\in {\mathcal{H}}_{\text{lin}}$ such that ${\mathbf{P}}_{(X,Y)~D}\left(Y=h^{*}(X)\right)=1$, and therefore ${\mathrm{er}}_D^{0-1}\left[{\mathcal{H}}_{\text{lin}}\right]=0$. Thus our goal is to show that ∃ a strictly increasing function $g:{ℝ}_{+}\to {ℝ}_{+}$ that is continuous at 0 with g(0) = 0 such that for all $f\in {\mathcal{F}}_{\text{spwlin}}$,

$${\mathrm{er}}_D^{0-1}[\mathrm{argmax}\circ \mathbf{f}]\le g\left({\mathrm{er}}_D^{\text{OvA},\log }[\mathbf{f}]-{\mathrm{er}}_D^{\text{OvA,log}}\left[{\mathcal{F}}_{\text{spwlin}}\right]\right).$$

We will do this in two parts:

1. We will show that ${\mathrm{er}}_D^{\text{OvA},\log }\left[{\mathcal{F}}_{\text{spwlin}}\right]=0$.

2. We will show that for all $f\in {\mathcal{F}}_{\text{spwlin}}$, ${\mathrm{er}}_D^{0-1}[\mathrm{argmax}\circ \text{f}]\le \frac{1}{\ln (2)}{\mathrm{er}}_D^{\text{OvA,log}}[\text{f}]$.

Putting these together will then give that for all $f\in {\mathcal{F}}_{\text{spwlin}}$,

$${\mathrm{er}}_D^{0-1}[\mathrm{argmax}\circ \mathbf{f}]\le \frac{1}{\ln (2)}\left({\mathrm{er}}_D^{\text{OvA},\log }[\mathbf{f}]-{\mathrm{er}}_D^{\text{OvA},\log }\left[{\mathcal{F}}_{\text{spwlin}}\right]\right).$$

Part 1.

We will show that for any sufficiently small $ϵ>0,\exists {\mathbf{f}}^{ϵ}\in {\mathcal{F}}_{\text{spwlin}}$ such that ${\mathrm{er}}_D^{\text{OvA},\log }\left[f^{ϵ}\right]<ϵ$; this will establish that ${\mathrm{er}}_D^{\text{OvA,}\log }\left[{\mathcal{F}}_{\text{spwlin}}\right]=0$.

Let 0 < ϵ < 2n ln(2). Since $h^{*}\in {\mathcal{H}}_{\text{lin}}$, we have $\exists {\left\{{\text{w}}_y^{*},b_y^{*}\right\}}_{y=1}^n$ such that

$$h^{*}(x)\in {\mathrm{argmax}}_{y\in [n]}{\left(w_y^{*}\right)}^{\top }x+b_y^{*}\forall x$$

Define $f^{*}\in {\mathcal{F}}_{\text{spwlin}}$ as

$$\begin{gathered} f_y^{*}(\text{x})=\underset{y^{\prime }\ne y}{\min }\left\{{\left({\mathbf{w}}_y^{*}-{\mathbf{w}}_{y^{\prime }}^{*}\right)}^{\top }\text{x}+\left(b_y^{*}-b_{y^{\prime }}^{*}\right)\right\} \\ =\underset{y^{\prime }\ne y}{\min }\left\{\left({\left({\mathbf{w}}_y^{*}\right)}^{\top }\mathbf{x}+b_y^{*}\right)-\left({\left({\mathbf{w}}_{y^{\prime }}^{*}\right)}^{\top }\mathbf{x}+b_{y^{\prime }}^{*}\right)\right\}. \end{gathered}$$

Then we have

$${\mathbf{P}}_{(X,Y)~D}\left(f_Y^{*}(X)>0\right)=1.$$

Therefore ∃κ > 0 such that

$${\mathbf{P}}_{(X,Y)~D}\left(f_Y^{*}(X)<\kappa \right)\le \frac{ϵ}{2n\ln (2)}.$$

Define ${\mathbf{f}}^{ϵ}\in {\mathcal{F}}_{\text{spwlin}}$ as

$$f_y^{ϵ}(x)=\frac{f_y^{*}(x)}{\kappa }\ln \left(\frac{1}{e^{ϵ/2n}-1}\right)\mathrm{.}$$

Then it can be verified that

$$f_y^{*}(\text{x})>0\Rightarrow f_y^{ϵ}(\text{x})>0\Rightarrow {\psi }_{\text{OvA},\log }\left(y,{\text{f}}^{ϵ}(\text{x})\right)\le n\ln (2),$$

and moreover,

$$f_y^{*}(\text{x})\ge \kappa \Rightarrow f_y^{ϵ}(\text{x})\ge \ln \left(\frac{1}{e^{ϵ/2n}-1}\right)\Rightarrow {\psi }_{\text{OvA},\log }\left(y,{\text{f}}^{ϵ}(\text{x})\right)\le \frac{ϵ}{2}.$$

This gives

$$\begin{gathered} {\mathrm{er}}_D^{\text{OvA},\log }\left[{\text{f}}^{ϵ}\right]={\mathbf{E}}_{(X,Y)~D}\left[{\psi }_{\text{OvA},\log }\left(Y,{\mathbf{f}}^{ϵ}(X)\right)\right] \\ \le {\mathbf{P}}_{(X,Y)~D}\left(0<f_Y^{*}(X)<\kappa \right)\cdot \mathbf{E}\left[{\psi }_{\text{OvA,log}}\left(Y,{\mathbf{f}}^{ϵ}(X)\right)\mid 0<f_Y^{*}(X)<\kappa \right] \\ +{\mathbf{P}}_{(X,Y)~D}\left(f_Y^{*}(X)\ge \kappa \right)\cdot \mathbf{E}\left[{\psi }_{\text{OvA,log}}\left(Y,{\mathbf{f}}^{ϵ}(X)\right)\mid f_Y^{*}(X)\ge \kappa \right] \\ \le \frac{ϵ}{2n\ln (2)}\cdot n\ln (2)+1\cdot \frac{ϵ}{2} \\ =ϵ. \end{gathered}$$

Part 2.

Let $\mathbf{f}\in {\mathcal{F}}_{\text{spwlin}}$, and let ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^n$ be such that

$$f_y(\mathbf{x})=\underset{y^{\prime }\ne y}{\min }\left\{{\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)\right\}\forall \text{x}.$$

Define $h:\mathcal{X}\to \mathcal{Y}$ such that

$$h(x)\in {\mathrm{argmax}}_{y\in [n]}{f}_y(x)\forall \text{x}$$

Then we have

$$\begin{gathered} {\mathrm{er}}_D^{0-1}[h]={\mathbf{E}}_{(X,Y)~D}\left[{\ell }_{0-1}(Y,h(X))\right] \\ ={\mathbf{E}}_{(X,Y)~D}[\mathbf{1}(h(X)\ne Y)] \\ ={\mathbf{E}}_{(X,Y)~D}\left[\sum _{y\ne Y}\mathbf{1}(h(X)=y)\right] \\ \le {\mathbf{E}}_{(X,Y)~D}\left[\sum _{y\ne Y}\mathbf{1}\left(f_y(X)\ge 0\right)\right]\text{(by}\text{definition}\text{of}h\text{and}\text{Lemma}\text{1)} \\ \le \frac{1}{\ln (2)}{\mathbf{E}}_{(X,Y)~D}\left[\sum _{y\ne Y}\ln \left(1+e^{f_y(X)}\right)\right] \\ \le \frac{1}{\ln (2)}{\mathbf{E}}_{(X,Y)~D}\left[\ln \left(1+e^{-f_Y(X)}\right)+\sum _{y\ne Y}\ln \left(1+e^{f_y(X)}\right)\right] \\ \left(\text{since}\ln \left(1+e^{-f_y(\mathbf{x})}\right)\ge 0\forall (\text{x},y)\right) \\ =\frac{1}{\ln (2)}{\mathbf{E}}_{(X,Y)~D}\left[{\ell }_{\text{OvA},\log }(Y,\mathbf{f}(X))\right] \\ =\frac{1}{\ln (2)}{\mathrm{er}}_D^{\text{OvA},\log }[\text{f}]. \end{gathered}$$

C. Proof of Theorem 3

Proof.

Let $w_1,\dots ,w_n\in {ℝ}^d$, $b_1,\dots ,b_n\in ℝ$, and let $f\in {\mathcal{F}}_{\text{spwlin}}$ be parametrized by ${\left\{{\mathbf{w}}_y,b_y\right\}}_{y=1}^n$, so that

$$f_y(\mathbf{x})=\underset{y^{\prime }\ne y}{\min }\left\{{\left({\mathbf{w}}_y-{\mathbf{w}}_{y^{\prime }}\right)}^{\top }\mathbf{x}+\left(b_y-b_{y^{\prime }}\right)\right\}\forall \mathbf{x}.$$

We will show that

$${\mathrm{argmax}}_{y\in [n]}{f}_y(\mathbf{x})={\mathrm{argmax}}_{y\in [n]}{\mathbf{w}}_y^{\top }\mathbf{x}+b_y;$$

this will establish the result.

To see that the above claim is true, notice that we can write

$$f_y(\mathbf{x})=\left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)-\underset{y^{\prime }\ne y}{\max }\left\{{\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right\}.$$

In other words, f_y(x) is the difference between $\left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)$ and the largest value of $\left({\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right)$ among y′ ≠ y. Clearly, this difference is largest when $\left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)\ge \left({\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right)\forall y^{\prime }\ne y$ (in particular, in this case the difference is non-negative; in all other cases, the difference is negative, and therefore smaller). Thus

$$f_y(\mathbf{x})\ge f_{y^{\prime }}(\mathbf{x})\forall y^{\prime }\ne y\iff \left({\mathbf{w}}_y^{\top }\mathbf{x}+b_y\right)\ge \left({\mathbf{w}}_{y^{\prime }}^{\top }\mathbf{x}+b_{y^{\prime }}\right)\forall y^{\prime }\ne y.$$

This proves the claim.

D. Proof of Corollary 4

This follows directly from the proof of Theorem 3.

E. Details of Real Data Sets Used in Experiments in Section 5.2

Table 3:

Multiclass classification data sets used in experiments in Section 5.2.

Data set	# train	# validation	# test	# classes (n)	# features (d)
Covertype (50K)	30000	10000	10000	7	54
Digits	5620	1874	3498	10	16
USPS	5468	1823	2007	10	256
MNIST (70K)	45000	15000	10000	10	780
CIFAR10	37500	12500	10000	10	3072
Sensorless	35105	11702	11702	11	48
Letter	10500	4500	5000	26	16

Notes:

Subsampling: For Covertype, we used a random subsample of the original data set containing 50,000 examples (the original data set has 581,012 examples).

Image data sets with pixel features: The versions of the USPS and MNIST datasets that we used came with features scaled to the ranges [−1, 1] and [0, 1], respectively. For CIFAR10, we similarly scaled the features to the range [0, 1] by dividing all features by 255.