Constrained optimization as ecological dynamics with applications to random quadratic programming in high dimensions

Abstract

Quadratic programming (QP) is a common and important constrained optimization problem. Here, we derive a surprising duality between constrained optimization with inequality constraints, of which QP is a special case, and consumer resource models describing ecological dynamics. Combining this duality with a recent ”cavity solution”, we analyze high-dimensional, random QP where the optimization function and constraints are drawn randomly. Our theory shows remarkable agreement with numerics and points to a deep connection between optimization, dynamical systems, and ecology.

I. INTRODUCTION

Optimization is an important problem for numerous disciplines, including physics, computer science, information theory, machine learning, and operations research [1–4]. Many optimization problems are amenable to analysis using techniques from the statistical physics of disordered systems [5–7]. Over the last few years, similar methods have been used to study community assembly and ecological dynamics suggesting a deep connection between ecological models of community assembly and optimization [8–16].Yet, the exact relationship between these two fields remains unclear.

Here, we show that constrained optimization problems with inequality constraints are naturally dual to an ecological dynamical system describing a generalized consumer resource model [17–19]. As an illustration of this duality, we focus on a particular important and commonly encountered constrained optimization problem: quadratic programming (QP) [1]. In QP, the goal is to minimize a quadratic objective function subject to inequality constraints. We show that QP is dual to one of the most famous models of ecological dynamics, MacArthur’s Consumer Resource Model (MCRM), a system of ordinary differential equations describing how species compete for a pool of common resources [17–19]. We also show that the Lagrangian dual of QP has a natural description in terms of generalized Lotka-Volterra equations that can be derived from the MCRM in the limit of fast resource dynamics.

We then consider random quadratic programming (RQP) problems where the optimization function and inequality constraints are drawn from a random distribution. We exploit a recent ”cavity solution” to the MCRM by one of us to construct a mean-field theory for the statistical properties of RQP [12]. Our theory is exact in infinite dimensions and shows remarkable agreement with numerical simulations even for moderately sized finite systems. This duality also allows us to use ideas from ecology to understand the behavior of RQP and interpret community assembly in the MCRM as an optimization problem.

II. OPTIMIZATION AS ECOLOGICAL DYNAMICS

We begin by deriving the duality between constrained optimization and ecological dynamics. Consider an optimization problem of the form

$$\begin{array}{ll}\underset{R}{\mathrm{minimize}}\hfill & f(R)\hfill \\ \text{subject}\text{to}\hfill & g_i(R)\le 0,i=1,\dots ,S\hfill \\ \hfill & R_{\alpha }\ge 0,\alpha =1,\dots ,M.\hfill \end{array}.$$ (1)

where the variables being optimized R = (Ri, R₂,…, R_m) are constrained to be non-negative. We can introduce a ‘generalized’ Lagrange multiplier λ_i for each of the S inequality constraints in our optimization problem. In terms of the λ_i, we can write a set of conditions collectively known as the Karush-Kuhn-Tucker (KKT) conditions that must be satisfied at any local optimum R_min of our problem [1–3]. We note that for this reason, in the optimization literature the λ_i are often called KKT-multipliers rather than Lagrange multipliers. The KKT conditions are:

Stationarity: ${\nabla }_{\text{R}}f\left(R_{\min }\right)+\sum _j{\text{λ}}_j{\nabla }_{\text{R}}{g}_j\left(R_{\min }\right)=0$

Primal feasibility: $g_i\left(R_{\min }\right)\le 0$

Dual feasibility: λ_i ≥ 0

Complementary slackness: ${\text{λ}}_i{g}_i\left(R_{\min }\right)=0,$

where the last three conditions must hold for all i = 1,…, M. The KKT conditions have a straightforward and intuitive explanation. At the optimum R_min, either g_i(R_min) = 0 and the constraint is active λ_i ≥ 0, or g_i(R_min) ≤ 0 and the constraint is inactive λ_i = 0. In our problem, the KKT conditions must be supplemented with the additional requirement of positivity R_α ≥ 0.

One can easily show that the four KKT conditions and positivity are also satisfied by the steady states of the following set of differential equations restricted to the space λ_i, R_α ≥ 0:

$$\begin{gathered} \frac{d{\text{λ}}_i}{dt}={\text{λ}}_i{g}_i(R) \\ \frac{d{R}_{\alpha }}{dt}=\left[-{\partial }_{R_{\alpha }}f(R)-\sum _j{\text{λ}}_j{\partial }_{R_{\alpha }}{g}_j(R)\right]R_{\alpha } \end{gathered}$$ (2)

The first of these equations just describes exponential growth of a “species” i with a resource-dependent “growth rate” g_i(R). Species with g_i(R_min) ≤ 0 correspond to constraints that are inactive and go extinct in the ecosystem (i.e, λ_{i min} = 0), whereas species with g_i(R_min) = 0 survive at steady state and correspond to active constraints with λ_{i min} ≠ 0 (see Fig. 1 for a simple two-dimensional example). The second equation in (2) performs a “generalized gradient descent” on the optimization function $f(R)+\sum _j{\text{λ}}_j{g}_j(R)$ (note the extra factor of R_a in our dynamics compared to the usual gradient descent equations). In the context of ecology, these equations describe the dynamics of a set of resources {R_α} produced at a rate $-{\partial }_{R_{\alpha }}f(R)R_{\alpha }$ and consumed by individuals of species j at a rate ${\text{λ}}_j{\partial }_{\alpha }{g}_j(R)R_{\alpha }.$

FIG. 1: Constrained optimization with inequality constraints is dual to an ecological dynamical system described by a generalized consumer resource model (MCRM).

The variables to be optimized (hexagons) and Lagrange multipliers (ovals) are mapped to resources and species respectively. Species must consume resources to grow. (Bottom left) A quadratic programming (QP) problem with two inequality constraints where the unconstrained optimum differs from the constrained optimum. (Bottom right) Dynamics for MacArthur’s Consumer Resource Model that is dual to this QP problem. The steady-state resource or species abundances correspond to the value of variables or Lagrange multipliers at the QP optimum. For this reason, species corresponding to inactive constraints go extinct.

This suggests a simple dictionary for constructing systems dual to optimization problems with inequality constraints (see Fig. 1) . The variables are resources whose dynamics are governed by the gradient of the function being optimized. Each inequality is associated with a species through its corresponding Lagrange (KKT) multiplier. Species that survive in the ecosystem correspond to active constraints, whereas species that go extinct correspond to inactive constraints. The steady-state values of the resource and species abundances correspond to the local optimum R_min and Lagrange multipliers at the optimum {λ_{j min}}, respectively. Finally, the f(R_min) are closely related to Lyapunov functions known to exist in the literature for specific choices of resource dynamics [15, 18, 19].

III. ECOLOGICAL DUALS OF QUADRATIC PROGRAMMING (QP)

For the rest of the paper, we focus on QP where the optimization function is quadratic, $f(R)=\frac{1}{2}{R}^{T}QR+b^{T}R,$ with Q a positive semidefinite matrix, and linear inequality constraints. The positivity of Q guarantees that the problem is convex. By going to the eigenbasis of Q, we can always rewrite the QP problem as minimizing a square distance

$$\begin{array}{ll}\underset{R}{\mathrm{minimize}}\hfill & \frac{1}{2}\Vert R-K{\Vert }^2\hfill \\ \text{subject}\text{to}\hfill & \sum _{\alpha }{c}_{i\alpha }{R}_{\alpha }\le m_i,i=1,\dots ,S\hfill \\ \hfill & R_{\alpha }\ge 0,\alpha =1,\dots ,M.\hfill \end{array}.$$ (3)

Using (2), we can construct the dual ecological model:

$$\begin{array}{l}\frac{d{\text{λ}}_i}{dt}={\text{λ}}_i\left(\sum _{\alpha }{c}_{i\alpha }{R}_{\alpha }-m_i\right)\\ \frac{d{R}_{\alpha }}{dt}=R_{\alpha }\left(K_{\alpha }-R_{\alpha }\right)-\sum _j{\text{λ}}_j{c}_{j\alpha }{R}_{\alpha }.\end{array}$$ (4)

The is the famous MacArthur Consumer Resource Model (MCRM) which was first introduced by MacArthur and Levins in their seminal papers [18, 20] and has played an extremely important role in theoretical ecology [21, 22].

In optimization problems, one often works with the Lagrangian dual of an optimization problem. We show in Appendix that the dual to (3) is just

$$\begin{array}{ll}\underset{{\text{λ}}_i}{\mathrm{maximize}}\hfill & \sum _i{\text{λ}}_i\left[{\kappa }_i-\frac{1}{2}\sum _j{\alpha }_{ij}{\text{λ}}_j\right]\hfill \\ \text{subject}\text{to}\hfill & {\text{λ}}_i\ge 0,\hfill \end{array}$$ (5)

with ${\kappa }_i={\sum }_{\alpha }{K}_{\alpha }\left(c_{i\alpha }-m_i\right),{\alpha }_{ij}={\sum }_{\alpha }{c}_{i\alpha }{c}_{j\alpha },$ and thesum restricted to α for which R_{α min} ≠ 0. It is once again straightforward to check that the local minima of this problem are in one-to-one correspondence with steady states of the Generalized Lotka-Volterra Equations (GLVs) of the form:

$$\frac{d{\text{λ}}_i}{dt}={\text{λ}}_i\left({\kappa }_i-\sum _j{\alpha }_{ij}{\text{λ}}_j\right)$$ (6)

As with the primal problem, the species in the GLV have a natural interpretation as Lagrange multipliers enforcing inequality constraints. This GLV can also be directly obtained from the MCRM in (4) in the limit where the resource dynamics are extremely fast by setting $\frac{d{R}_{\alpha }}{dt}=0$ in the second equation and plugging in the steady-state resource abundances into the first equation [18, 19] (see Appendix). This shows the Lagrangian dual of QP maps to a dynamical system described by a GLV -which itself can be derived from the MCRM which is the dynamical dual to the primal optimization problem!

IV. RANDOM QUADRATIC PROGRAMMING (RQP)

Recently, the MCRM was analyzed in the high-dimensional limit where the number of resources and species in the regional species pool is large (S, M ≫ 1). In this limit, the resource dynamics were extremely complex, with many resources deviating significantly from their unperturbed values and a large fraction of species in the regional pool going extinct [12]. In terms of the corresponding optimization problem, this suggests that f (R_min) will generically be far from zero and many of constraints will be inactive.

To better understand this, we analyzed Random quadratic programming (RQP) problems in high dimension. In RQP, the parameters in (3) are drawn from random distributions (see Fig. 2A). We focus on the case where the K_α and m_i are independent random normal variables drawn from Gaussians with means K and m and variances ${\sigma }_K^2\text{and}{\sigma }_m^2,$ respectively. The elements of the constraint matrix c_iα are also drawn from Gaussians with mean μ_c/M and variance ${\sigma }_c^2/M$ [35].This scaling with M is necessary to ensure that the sum that appears in the inequality constraints in (3) has a good thermodynamic limit when M, S → ∞ with $M/S=\gamma$ held fixed.

FIG. 2: Random Quadratic Programming (RQP).

(a) In RQP, the parameters of the quadratic optimization function and inequality constraints are drawn from a random distribution. Effect of varying the ratio of constraints to variables S/M on (b) the value of the optimization function f(R_min)/M, (c) the fraction of non-zero variables $\frac{M^{*}}{M}$ and (d) the fraction of active constraints $\frac{S^{*}}{S}.$ Cavity solutions are solid lines and shaded region show ±1 standard deviation from 50 independent optimizations of RQP using the CVX-OPT package in Python 3 with M = 100, μ_c = 1, K = 1, σ_K = 1, m = 1, σ_m = 0:1. Code is available in supplementary files.

We are especially interested in understanding the statistical properties of solutions to the RQP (see Fig. 2A) . Among the quantities we examine are the expectation value of the optimized function at the minima $\langle f\left(R_{\min }\right)\rangle /M,$ the fraction of active constraints, S*/S, the fraction of variables that are non-zero at the optimum, M* /M, as well the first two moments of R_αmin and λ_{j min} (see Appendix for details).

It is possible to a derive mean-field theory (MFT) for the statistical properties of the optimal solution in the RQP -or correspondingly the steady-states of the MCRM -using the cavity method. The basic idea behind the cavity method is to derive self-consistency equations that relate the optimization problem (ecosystem) with M +1 variables (resources) and S +1 inequality constraints (species) to a problem where a constraint (species) and variable (resource) have been removed: (M + 1, S + 1) → (M, S) [12]. The need to remove both a constraint and variable is important for keeping all order one terms in the thermodynamic limit [23, 24]. In what follows, we focus on the replica-symmetric solution.

The cavity equation exploits the observations the constraint ${\sum }_{\alpha =1}^M{c}_{i\alpha }{R}_{\alpha }$ is a sum of many random variables, c_iα. When M ≫ 1 , due to the law of large numbers we can model such a sum by a random variable drawn from a Gaussian whose mean and variance involve the statistical quantities described above. Less obvious from the perspective of QP is that we need to introduce a second mean-field quantity $K_{\alpha }^{eff}$ (see Appendix and [12]). After introducing the Lagrange multipliers that enforce the inequality constraints, the optimization function to be minimized takes the form

$$\frac{1}{2}\Vert R-K{\Vert }^2+\sum _j{\text{λ}}_j\left(c_{j\alpha }{R}_{\alpha }-m_j\right)=\frac{1}{2}\sum _{\alpha }\left\{R_{\alpha }\left[R_{\alpha }-K_{\alpha }^{eff}(\text{λ})\right]+K_{\alpha }\left[K_{\alpha }-R_{\alpha }\right]\right\},$$

where we have defined the mean-field variable

$$K_{\alpha }^{eff}(\text{λ})=K_{\alpha }-\sum _{j=1}^S{\text{λ}}_j{c}_{j\alpha }.$$

Since $K_{\alpha }^{eff}(\text{λ})$ is also a sum of many terms containing c_iα, it can also be approximated as a random variable drawn from a Gaussian whose mean and variance are calculated self-consistently.

The full derivation of the replica symmetric mean-field equations is identical to that in [12] and is given in the Appendix. The resulting self-consistent mean-field cavity equations can be solved numerically in Mathematica. Fig. 2 shows the results of our mean-field equations and comparisons to numerics where we directly optimize the RQP problem over many independent realizations using the CVXOPT package in Python [25]. Notice the remarkable agreement between our MFT and results from direct optimization even for moderate system sizes with M = 100. In the Appendix, we show that the cavity solution can also accurately describe the dual MCRM.

Fig. 2 also shows that the statistical properties of the QP solutions change as we vary the number of constraints S and the variance of the constraint matrix c_iα. When S ≪ M, the expectation value of the optimization function f (R_min)/M approaches zero -the minimum for the unconstrained problem. In this limit, the few constraints that are present are also active. As S/M is increased, the fraction of active constraints quickly drops, f (R_min)/M quickly increases, after which both quantities reach a plateau where they vary very slowly with S. The value of the the plateau depends on a_c. Increasing the variance of the constraints results in more active constraints and a larger value of f (R_min) at the optimum.

These results about RQP can be naturally understood using ideas from ecology. Intuitively, a smaller σ_c means more “redundant” constraints. In ecology, this is the principle of limiting similarity: species with large niche overlaps (similar c_i ) competitively exclude each other [18–22]. In the language of optimization, this ecological intuition suggests that when constraints are similar enough, only the most stringent of these will be active due to an effective competitive exclusion between constraints. Thus, in RQP competitive exclusion becomes a statement about the geometry of how random planes in high dimension repel each other at the corners of simplices. In all cases, increasing S increases the total number of active constraints (species) even though the fraction of active constraints decreases. For this reason, the optimization problem is more constrained for larger S and f (R_min)/M is larger. Finally the plateau in statistical quantities at large S can be understood as arising from what in ecology has been called “species packing” -there is a capacity to the number of distinct species that any ecosystem can typically support [18, 20].

V. DISCUSSION

In this paper, we have derived a surprising duality between constrained optimization problems and ecologically inspired dynamical systems. We showed that QP (in any dimension) maps to one of the most famous models of ecological dynamics, MacArthur’s Consumer Resource Model (MCRM) -a system of ordinary differential equations describing how species compete for a pool of common resources. By combining this mapping with a recent ‘cavity solution’ to the MCRM, we constructed a mean-field theory for the statistical properties of RQP that showed remarkable agreement with numerical simulations. Intuitions from ecology suggest that the geometry of constrained optimization can be described using a competitive exclusion between constraints which in our case correspond to random high-dimensional hyper-planes. This work suggests that the deep connection between geometry, ecology, and high-dimensional random ecosystems is a generic property of a large class of generalized consumer resource models [26]. Our works also gives a natural explanation of the existence of Lyapunov functions in these models. Many of these ideas can also be generalized to settings that result in asymmetric interactions between species [27].

We have focused on convex QP, where the quadratic form in the objective function is positive semi-definite. When applied to indefinite QP, our mapping no longer produces a physically feasible ecosystem. In this more general scenario, the KKT conditions themselves are necessary but not sufficient for global optimality (see, for example, [1]). Several algorithms were proposed recently to tackle non-convex QP in the optimization community [28, 29]. This problem is known to be NP-complete [30], prompting recent interest in approaches based on quantum annealing or adiabatic quantum computation. [31, 32].

Our results on the duality between a generic consumer resource model and quadratic programming extend recent works on more specialized, fine-tuned consumer resource models studies in [15, 33]. The specialized consumer resource models studied in these works exhibited a strict phase-transition between a regime where all resources were pinned to the same value and M species survived (what the authors called a shielded phase) and a phase where S* < M species survived in the ecosystem (what the authors called a vulnerable phase). In [33], it was pointed out that this transition is reminiscent of constraint satisfaction problems and in particular random linear programming [34]. The models studied in our paper do not exhibit this phase transition because far less stringent assumptions are made on resource dynamics. Nonetheless, our works shows that even in generic models resource models the fraction of active constraints saturates, a behavior reminiscent of the shielded phase studied in [15, 33]. Furthermore, by explicitly constructing a general duality between a very large set of consumer resource models and constrained optimization problems, our work makes clear how we can relate the idea of active constraints in optimization theory to ideas in ecological dynamics.

Our works also suggests a simple way of speeding up simulations for steady-states of consumer resource models. By mapping the ecological dynamics onto convex optimization, we can make use of powerful numerical and computational techniques to calculate for steady-state properties of consumer resource models.

FIG. 3: Comparison of Cavity Solution (solid line), RQP (long dash line), and dual MCRMs (short dash line).

The simulations represent averages from 50 independent realizations and parameters as in Fig. 2 of main text.