Algorithm 1.

Sample-Feature Selection (SFS)

	Input: Regularization scheme X, hyperparameter λ (and r for partial regularization), number of desired features τ, number of desired samples k− and k+, and threshold ϵ.
	Initialization: Let $({\mathit{\beta }}^{(0)},{\beta }_0^{(0)})\in {ℝ}^p\times ℝ$ be arbitrarily chosen such that ${\Vert {\mathit{\beta }}^{(0)}\Vert }_0=\tau$. Set t = 0 and f(0) = ∞.
1:	repeat	▹ Beginning of BCD
2:	Update sample weights α(t+1):
	2.1: $l_i\leftarrow \sigma \left({\mathbf{y}}_i\left({\left({\mathbf{x}}_i\right)}^{\top }{\mathit{\beta }}^{(t)}+{\beta }_0^{(t)}\right)\right)$
	2.2: $\left\{i_1^{+},\dots ,i_{|I(+1)|}^{+}\right\}\leftarrow \text{Sort}$ ‘+1’ samples, s. t. $l_{i_j^{+}}+\le l_{i_{j+1}^{+}}$
	2.3: $I^{+}\leftarrow \left\{i_1^{+},\dots ,i_{k_{+}}^{+}\right\}$
	2.4: $\left\{i_1^{-},\dots ,i_{|I(-1)|}^{-}\right\}\leftarrow \text{Sort}$ ‘−1’ samples, s. t. $l_{i_j^{-}}+\le l_{i_{j+1}^{-}}$
	2.5: $I^{-}\leftarrow \left\{i_1^{-},\dots ,i_{k_{-}}^{-}\right\}$
	2.6: ${\alpha }_i^{(t+1)}\leftarrow \{\begin{array}{l}1,\text{if}i\in I^{+}\cup I^{-}\hfill \\ 0,\text{otherwise}\hfill \end{array}$
3:	Update features weights and bias $\left({\mathit{\beta }}^{(t+1)},{\beta }_0^{(t+1)}\right)$ via Algorithm 2 (X = H) or Algorithm 3 (X = P)
4:	Update $f^{(t+1)}\leftarrow f_X\left({\mathit{\alpha }}^{(t+1)},{\mathit{\beta }}^{(t+1)},{\beta }_0^{(t+1)}\right)$
5:	t ← t + 1
6:	until $\min \left(\frac{\left|f^{(t)}-f^{(t-1)}\right|}{\left|f^{(t)}\right|},\left|f^{(t)}\right|\right)<ϵ$	▹ Stopping Criteria
	Output: $\left(\widehat{\mathit{\alpha }},\widehat{\mathit{\beta }},{\widehat{\beta }}_0\right)=\left({\mathit{\alpha }}^{(t)},{\mathit{\beta }}^{(t)},{\beta }_0^{(t)}\right)$