On cheap entropy-sparsified regression learning

Abstract

Regression learning is one of the long-standing problems in statistics, machine learning, and deep learning (DL). We show that writing this problem as a probabilistic expectation over (unknown) feature probabilities – thus increasing the number of unknown parameters and seemingly making the problem more complex—actually leads to its simplification, and allows incorporating the physical principle of entropy maximization. It helps decompose a very general setting of this learning problem (including discretization, feature selection, and learning multiple piece-wise linear regressions) into an iterative sequence of simple substeps, which are either analytically solvable or cheaply computable through an efficient second-order numerical solver with a sublinear cost scaling. This leads to the computationally cheap and robust non-DL second-order Sparse Probabilistic Approximation for Regression Task Analysis (SPARTAn) algorithm, that can be efficiently applied to problems with millions of feature dimensions on a commodity laptop, when the state-of-the-art learning tools would require supercomputers. SPARTAn is compared to a range of commonly used regression learning tools on synthetic problems and on the prediction of the El Niño Southern Oscillation, the dominant interannual mode of tropical climate variability. The obtained SPARTAn learners provide more predictive, sparse, and physically explainable data descriptions, clearly discerning the important role of ocean temperature variability at the thermocline in the equatorial Pacific. SPARTAn provides an easily interpretable description of the timescales by which these thermocline temperature features evolve and eventually express at the surface, thereby enabling enhanced predictability of the key drivers of the interannual climate.

The problem of regression learning is one of the oldest questions in data science, seeking for relationships between the D explanatory feature information X _t  = (X _{1, t} ,…,X _{D, t} ) and the predicted M-dimensional real-valued variable Y _t  = (Y _{1, t} ,…,Y _{M, t} ) in the available statistics of data pairs {X _t ,Y _t }, with t = 1, …, T. In its most popular linear formulation, this relationship is described by means of a linear function and implemented as a convex numerical inference of a priori unknown M × D matrix of regression coefficients Λ from:

$$\begin{array}{cc}\hfill Y_{m,t}& ={\sum }_{d=1}^D{\mathrm{\Lambda }}_{m,d}{X}_{d,t}+\mathrm{noise},m=1,\cdots ,M.\hfill \end{array}$$ [1]

Identification of the relevant/significant subset of features is usually performed by the l ₂, l ₁, elastic net (linear combination of l ₂ and l ₁), or garrote-like sparsification, that attempt to force the elements of regression coefficient matrix Λ to zero.

Nonlinear extensions of this regression problem constitute the foundations of modern deep learning, aiming at the inference of superpositions ϕ ° ϕ ° ϕ(…) for generalized functions ϕ (e.g., perceptrons) with multiple linear regressions Eq. 1 , by means of first-order numerical algorithms like the stochastic gradient descent (SGD). This learning approach dwells on the mathematical proof of the thirteenth Hilbert problem by Arnold and Kolmogorov (1, 2). This proof and further results guarantee that any arbitrary continuous mapping from D-dimensional variable X _t to Y _t can be learned as an appropriate superposition ϕ ° ϕ ° ϕ(…) of continuous one-dimensional (1D) functions.

Despite its generality and wide acceptance, this common paradigm of regression learning bears several limitations, specifically: (i) popular feature selection approaches in regression learning, for example, l ₂, l ₁, elastic net, and garrote-like regularizations (3, 4) are characterized by the polynomial growth C(D)=c D ^a of computational and memory costs C(D) with feature dimension D, for exponent a being significantly larger than 1; ii) these approaches do not allow a direct probabilistic interpretation—consequently prohibiting the application of very general and cheap probabilistic feature selection paradigms from physics and information theory, like, for example, the maximum entropy principle; iii) the nonlinear extensions of regression learning in DL, that dwell on universal approximations by means of regression superpositions ϕ ° ϕ ° ϕ(…) of generalized linear functions ϕ, are characterized by intractability of higher-order derivatives (beyond the first derivative) of the loss function. In particular, issue (iii) results in a very limited applicability of second- and higher-order numerical methods (like the generalized Newton and interior point optimization methods) that could provide much higher convergence speed rates than the first-order numerical algorithms (like SGD). In many realistic learning applications, gradient descent (GD) and SGD exhibit extremely slow convergence rates, taking hundreds of thousands of expensive iterations, not allowing any mathematically justified stopping criteria and requiring high-performance computing (HPC) facilities.

The central message of this brief report is in showing that these three challenges (i) to (iii) can be addressed by making the regression problem seemingly more complex and extending the set of tunable parameters with a vector of D feature probabilities w = (w ₁,…,w _D ). In the linear case, this allows formulating a regression as an expectation value with respect to this unknown feature probability distribution:

$$\begin{array}{cc}\hfill Y_t& ={\mathbf{E}}_w\left[\mathrm{\Lambda },X_t\right]+\mathrm{noise}={\sum }_{d=1}^D{w}_d{\mathrm{\Lambda }}_{:,d}{X}_{d,t}+\mathrm{noise}.\hfill \end{array}$$ [2]

The common linear regression Eq. 1 then becomes a particular case of this more general probabilistic regression ( Eq. 2 ), with w being a uniform distribution w = (1/D,…,1/D). Generalization to nonlinear regression learning can be also achieved in the same probabilistic way and actually simplifies the problem, as will be shown below: Instead of dwelling on the thirteenth Hilbert problem and on superposition ϕ ° ϕ ° ϕ(…) of perceptrons (or other similar generalized linear model functions) that can lead to intractable high-order derivatives and slow convergence of GD and SGD, one can formulate the general nonlinear regression as a problem of a piece-wise linear learning of K linear regressions, coupled to entropic feature sparsification (5 –8)—and confined to K feature space discretization boxes with centers C _{:, k}  = (C _{1, k} ,…,C _{D, k} )^⊤, with the probability γ _{k, t} for a feature vector X _t to belong to the discretization box k (9). Such piece-wise linear probabilistic regression learning can be implemented as a numerical minimization of the following function:

$$\begin{array}{ll}L(w,\Lambda ,\gamma ,C)=\hfill & \underset{\text{loss of feature discretization in K boxes}}{\underbrace{\frac{1}{TK}{\displaystyle \sum }_{t,d,k}^{T,D,K}{w}_d{\gamma }_{k,t}{\left(X_{d,t},-,C_{d,k}\right)}^2}}\hfill \\ +{ϵ}_{l_2}\hfill & \underset{\text{loss}\hspace{0.17em}\hspace{0.17em}\text{of}\hspace{0.17em}\hspace{0.17em}{\text{l}}_2\hspace{0.17em}\hspace{0.17em}\text{regression regularization}}{\underbrace{{\displaystyle \sum }_{m,d,k=1}^{M,D,K}{\Lambda }_{d,m,k}^2}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{ϵ}_w\underset{\text{loss}\hspace{0.17em}\hspace{0.17em}\text{of}\hspace{0.17em}\hspace{0.17em}\text{entropic}\hspace{0.17em}\hspace{0.17em}\text{feature}\hspace{0.17em}\hspace{0.17em}\text{sparsification}}{\underbrace{{\displaystyle \sum }_{d=1}^D{w}_d\log w_d}}\hfill \\ +{ϵ}_r\hfill & \underset{\text{loss}\hspace{0.17em}\hspace{0.17em}\text{of}\hspace{0.17em}\hspace{0.17em}\text{K}\hspace{0.17em}\hspace{0.17em}\text{box-wise}\hspace{0.17em}\hspace{0.17em}\text{linear}\hspace{0.17em}\hspace{0.17em}\text{regressions}\hspace{0.17em}\hspace{0.17em}\text{approximations}}{\underbrace{\frac{1}{TM}{\displaystyle \sum }_{t,m,k}^{T,M,K}{\gamma }_{k,t}{\left(Y_{m,t},-{\text{,Λ}}_{0,m,k},-,{\displaystyle \sum }_{d=1}^D,w_d{,\Lambda }_{d,m,k},X_{d,t}\right)}^2}}.\hfill \end{array}$$ [3]

Learning the regression model parameters (w,Λ,γ,C) from a given set of data pairs {X _t ,Y _t }, t = 1, …, T, can be achieved with the monotonically convergent Sparse Probabilistic Approximation for Regression Task Analysis (SPARTAn) algorithm, reported in SI Appendix with the complete problem formulation. For any fixed set of four parameters (K,ϵ _r ,ϵ _w ,ϵ _{l 2} ), problem ( Eq. 3 ) can be solved analytically for each of the variables (Λ,γ,C), while keeping the remaining variables fixed. By applying the Euler–Lagrange principle to Eq. 3 , one obtains a convex optimization problem for the variable w, with a cheaply computable matrix of second derivatives. It appears that, for D ≤ 2, this problem allows an analytical solution for w, given by the Lambert W-functions that are widely used in quantum gravitation theory, solution of the Schrodinger equation, in polymer physics, and in material science (10). For D >  2, this convex problem can be solved numerically, adjusting the interior point Panua-Ipopt algorithm (11) to allow use of the sparsity and low-rank structure of the Hessian in problem ( Eq. 3 ). As can be seen from Fig. 1 A and B , this structural adaptation of the algorithm plays a crucial role in achieving the sublinear scaling of the overall computational and memory costs: Although application of common optimizers (like the commercial fmincon-solver from MathWorks and the general-purpose Ipopt algorithm) is characterized by the polynomial cost scaling C(D)=c D ^a (with a >  2 for computational cost and a >  1.3 for memory scaling), Panua-Ipopt adapted to the low-rank and sparsity patterns of the Hessian in ( Eq. 3 ) has a <  1 both for the overall memory and computational complexity scalings. Practically, it means that solving ( Eq. 3 ) for w, with one thousand data points T and one million feature data dimensions D—a typical situation in many realistic applications—requires only 13 min and around 16 GB of memory on a commodity laptop with 8 cores, against the roughly 336 y and 30 TB required by state-of-the-art optimization tools.

Fig. 1.

Comparison of SPARTAn to common tools for synthetic model data (A–E) and for the predictions of NINO3.4 from sea surface and bulk ocean temperatures (F–K), for the mean squared prediction error (MSE) and other prediction quality metrics.

Fig. 1A–E summarizes numerical experiments comparing SPARTAn to common regression learning and feature selection tools on randomly generated synthetic data sets from the very popular scalable (in a broad range of dimensions D and statistics sizes T) regression learning example 3 from Tibshirani’s seminal paper on l1 regularization (3). All of the compared algorithms are provided with the same information and run with the same hardware and software; convergence tolerance t o l was set to 10⁻¹² for all of the compared methods. Fifty cross-validations were performed in every experiment with every learning tool, to visualize the obtained average performances in Fig. 1 D and E . These results demonstrate that SPARTAn allows a marked and robust improvement of parameter identification quality, improved sparsity pattern detection, and low computational costs for very high-dimensional regression learning problems. Please note that the regression parameters in SPARTAn are learned as products with feature probabilities w _d , i.e., α _d  = w _d Λ _d . Uncertainty of α _d for a fixed w _d is expressed as Var(α _d )=w _d ²Var(Λ _d ), i.e., factor 10 reduction of feature probability w _d results in factor 100 reduction of variance for estimated α _d (Fig. 1C ).

Next, we consider a learning problem from climate research aiming at predictions of the El Niño Southern Oscillation (ENSO) encoded by the NINO3.4 climate index—a climate phenomenon governing the interannual variation of sea surface temperatures in the Pacific Ocean. The periodicity of this phenomenon is highly irregular, but its prediction has been gaining growing importance recently due to its recognized major impact on global climate and economy (12). Predictions of the NINO3.4 index with the modern AI tools are currently getting an increasing amount of attention both in geosciences and in the AI/ML communities (13, 14). The considered predicted variable Y _t represents a historical sequence of deseasonalized monthly NINO3.4 index values. Feature values X _t represent a series of dominant deseasonalized climate proxies X _t in the months preceding the Y _t , consistent of the 100 dominant Essential Orthogonal Function projections (EOFs) of the global sea surface temperatures resimulated by the National Oceanic and Atmospheric Administration in the United States and the 100 dominant ACCESS-O resimulated global sea surface and deep ocean water temperature EOFs in the equatorial Pacific Ocean. A detailed explanation of this data set is provided in (7).

To achieve a fair comparison and to avoid overfitting, in Fig. 1F–K , we deployed a multiple cross-validation procedure for comparison of SPARTAn to common regression tools when all of the methods were trained, validated, and tested using the same historical data. Each of the models was trained for a range of meta-parameters (for example, DL LSTM algorithm was repeatedly trained with various network architectures and various numbers of neurons in the hidden layer, and SPARTAn was trained with various values of parameters (K,ϵ _r ,ϵ _w ,ϵ _{l 2} )); details about trained meta-parameter ranges are available in the README-file (see the link in Data, Materials, and Software Availability section). Then, best-performing meta-parameters for each of the methods were determined on the validation data and, finally, the best performers from each method family were compared in Fig. 1F–H for the test historical data between 1992 and 2007, that were used neither in training nor in validation.

In Fig. 1 F–H , we see that SPARTAn comprehensively outperforms a wide range of machine learning algorithms, including the CNN (LSTM) that was best performing on training and validation data, in predicting the NINO3.4 index on test data. We also observe in Fig. 1G that SPARTAn is able to utilize additional subsurface ocean information at lead times of 5 mo, with a twofold reduction in MSE from > 0.012 to just over 0.006. None of the other methods show comparable improvements in skill, and only DL LSTM is able to approach but not match the low MSE values of SPARTAn even out to 15-mo lead times. In Fig. 1 I and J , the relative contributions of the surface temperature (SST) and subsurface vertical temperature gradients (dT/dz)—a proxy for thermocline variability—are clearly demonstrated with subseasonal predictability largely determined by surface temperatures (SST EOF 2 in Fig. 1I ) and where predictability beyond a season is strongly dependent on the stability of the thermocline (dT/dz EOF mode 3 in Fig. 1J ). That the coherent structures of the relevant modes are captured in the SPARTAn features is reflected in the weights shown in Fig. 1K .

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Code reproducing the comparison of numerical solvers in Fig. 1 A and B is available at https://github.com/goghino/scalable_regression. Data and code reproducing the comparison of NINO3.4 regression learning and prediction results (in Fig. 1 F–K ) are provided for academic use at https://github.com/horenkoi/SPARTAn. All study data are included in the article and/or SI Appendix .