Table 1.

Prominent options for choosing loss function and regularizer in feature extraction algorithms

Name	Loss function (L)	Regularizer (R)
AIC/BIC	∥y−〈ω,x〉∥2	∥ω∥0
Lasso	∥y−〈ω,x〉∥2	∥ω∥1
Elastic Net	∥y−〈ω,x〉∥2	$\parallel \omega {\parallel }_2^2$ + ∥ω∥1
Regularized Least Absolute
Deviations Regression	∥y−〈ω,x〉∥1	∥ω∥1
Classic SVM	max(0,1−y〈ω,x〉)a	$\frac{1}{2}\parallel \omega {\parallel }_2^2$
ℓ 1-SVM	max(0,1−y〈ω,x〉)a	$\frac{1}{2}\parallel \omega {\parallel }_1$
Logistic Regression	log(1+exp(−y〈ω,x〉))	$\frac{1}{2}\parallel \omega {\parallel }_1$

^*This is the so called Hinge loss

The ℓ ₁- and ℓ ₂-norm of a vector z=(z ₁,…,z _d)∈ℝ ^d are defined by $\parallel z{\parallel }_1=\sum _{j=1}^d\left|z_i\right|$ and $\parallel z{\parallel }_2={\left(\sum _{j=1}^d\right|z_i{|}^2)}^{1/2}$, respectively. The “ ℓ ₀-norm” ∥z∥₀, simply counts the number of non-zero entries of z