Consistent Second-Order Conic Integer Programming for Learning Bayesian Networks

Abstract

Bayesian Networks (BNs) represent conditional probability relations among a set of random variables (nodes) in the form of a directed acyclic graph (DAG), and have found diverse applications in knowledge discovery. We study the problem of learning the sparse DAG structure of a BN from continuous observational data. The central problem can be modeled as a mixed-integer program with an objective function composed of a convex quadratic loss function and a regularization penalty subject to linear constraints. The optimal solution to this mathematical program is known to have desirable statistical properties under certain conditions. However, the state-of-the-art optimization solvers are not able to obtain provably optimal solutions to the existing mathematical formulations for medium-size problems within reasonable computational times. To address this difficulty, we tackle the problem from both computational and statistical perspectives. On the one hand, we propose a concrete early stopping criterion to terminate the branch-and-bound process in order to obtain a near-optimal solution to the mixed-integer program, and establish the consistency of this approximate solution. On the other hand, we improve the existing formulations by replacing the linear “big-$M$” constraints that represent the relationship between the continuous and binary indicator variables with second-order conic constraints. Our numerical results demonstrate the effectiveness of the proposed approaches.

1. Introduction

A Bayesian network (BN) is a probabilistic graphical model consisting of a labeled directed acyclic graph (DAG) $𝒢=(V,E)$, in which the vertex set $V=\left\{V_1,\dots ,V_m\right\}$ corresponds to $m$ random variables, and the edge set $E$ prescribes a decomposition of the joint probability distribution of the random variables based on their parents in $𝒢$. The edge set $E$ encodes Markov relations on the nodes in the sense that each node is conditionally independent of its non-descendents given its parents. BNs have been used in knowledge discovery (Spirtes et al., 2000; Chen et al., 2019), classification (Aliferis et al., 2010), feature selection (Gao et al., 2015), latent variable discovery (Lazic et al., 2013) and genetics (Ott et al., 2004). They also play a vital part in causal inference (Pearl, 2009).

In this paper, we propose reformulations of the mixed-integer quadratic programs (MIQP) for learning the optimal DAG structure of BNs given $n$ continuous observations from a system of linear structural equation models (SEMs). While there exist exact integer-programming (IP) formulations for learning DAG structure with discrete data (Cussens, 2010, 2011; Hemmecke et al., 2012; Studenỳ and Haws, 2013; Barlett and Cussens, 2013; Oates et al., 2016a,b; Bartlett and Cussens, 2017; Cussens et al., 2017a,b), the development of tailored computational tools for learning the optimal DAG structure from continuous data has received less attention. In principle, exact methods developed for discrete data can be applied to continuous data. However, such methods result in exponentially sized formulations in terms of the number of binary variables. A common practice to circumvent the exponential number of binary variables is to limit the in-degree of each node (Cussens, 2011; Cussens et al., 2017b; Bartlett and Cussens, 2017). But, this may result in sub-optimal solutions. On the contrary, MIQP formulations for learning DAGs corresponding to linear SEMs require a polynomial number of binary variables. This is because for BNs with linear SEMs, the score function — i.e., the penalized negative log-likelihood (PNL) — can be explicitly written as a function of the coefficients of linear SEMs (Shojaie and Michailidis, 2010; van de Geer and Bühlmann, 2013; Park and Klabjan, 2017; Manzour et al., 2021). In contrast to the existing MIQPs (Park and Klabjan, 2017; Manzour et al., 2021), our reformulations exploit the convex quadratic objective and the relationship between the continuous and binary variables to improve the strength of the continuous relaxations.

Continuous BNs with linear SEMs have witnessed a growing interest in the statistics and computer science communities (van de Geer and Bühlmann, 2013; Raskutti and Uhler, 2018; Loh and Bühlmann, 2014; Ghoshal and Honorio, 2017; Solus et al.). In particular, it has been shown that the solution obtained from solving the PNL augmented by ${\ell }_0$ regularization, which introduces a penalty on the number of non-zero arc weights in the estimated DAG, achieves desirable statistical properties (Peters and Bühlmann, 2013; van de Geer and Bühlmann, 2013; Loh and Bühlmann, 2014). Moreover, if the model is identifiable (Peters and Bühlmann, 2013; Loh and Bühlmann, 2014), that is when the true causal graph can be identified from the joint distribution, then such a solution is guaranteed to uncover the true causal DAG when the sample size $n$ is large enough. However, given the difficulty of obtaining exact solutions, existing approaches for learning DAGs from linear SEMs have primarily relied on heuristics, using techniques such as coordinate descent (Fu and Zhou, 2013; Aragam and Zhou, 2015; Han et al., 2016) and non-convex continuous optimization (Zheng et al., 2018). Unfortunately, these heuristics are not guaranteed to achieve the desirable properties of the global optimal solution. Moreover, it is difficult to evaluate the statistical properties of a sub-optimal solution with no optimality guarantees (Koivisto, 2006). To bridge this gap, in this paper we develop mathematical formulations for learning optimal BNs from linear SEMs using a PNL objective with ${\ell }_0$ regularization. By connecting the optimality gap of the mixed-integer program to the statistical properties of the solution, we also establish an early stopping criterion under which we can terminate the branch-and-bound procedure and attain a solution which asymptotically recovers the true parameters with high probability.

Our work is related to recent efforts to develop exact tailored methods for DAG learning from continuous data. Xiang and Kim (2013) show that $A^{*}$-lasso algorithm tailored for DAG structure learning from continuous data with ${\ell }_1$-regularization, which introduces a penalty on the sum of absolute values of the arc weights, is more effective than the previous approaches based on dynamic programming (e.g., Silander and Myllymäki, 2006) that are suitable for both discrete and continuous data. Park and Klabjan (2017) develop a mathematical program for DAG structure learning with ${\ell }_1$ regularization. Manzour et al. (2021) improve and extend the formulation by Park and Klabjan (2017) for DAG learning from continuous data with both ${\ell }_0$ and ${\ell }_1$ regularizations. The numerical experiments by Manzour et al. (2021) demonstrate that as the number of nodes grows, their MIQP formulation outperforms $A^{*}$-lasso and the existing IP methods; this improvement is both in terms of reducing the IP optimality gap, when the algorithm is stopped due to a time limit, and in terms of computational time, when the instances can be solved to optimality. In light of these recent efforts, the current paper makes important contributions to this problem at the intersection of statistics and optimization.

• The statistical properties of optimal PNL with ${\ell }_0$ regularization have been studied extensively (Loh and Bühlmann, 2014; van de Geer and Bühlmann, 2013). However, it is often difficult to obtain an optimal solution and no results have been established on the statistical properties of approximate solutions. In this paper, we give an early stopping criterion for the branch-and-bound process under which the approximate solution gives consistent estimates of the true coefficients of the linear SEM. Our result leverages the statistical consistency of the PNL estimate with ${\ell }_0$ regularization (van de Geer and Bühlmann, 2013; Peters and Bühlmann, 2013) along with the properties of the branch-and-bound method wherein both lower and upper bound values on the objective function are available at each iteration. By connecting these two properties, we obtain a concrete early stopping criterion, as well as a proof of consistency of the approximate solution. To the best of our knowledge, this result is the first of its kind for DAG learning.

• In spite of recent progress, a key challenge in learning DAGs from linear SEMs is enforcing bounds on arc weights. This is commonly modeled using the standard “big-$M$ constraint” approach (Park and Klabjan, 2017; Manzour et al., 2021). As shown by Manzour et al. (2021), this strategy leads to poor continuous relaxations for the problem, which in turn results in slow lower bound improvement in the branch-and-bound tree. In particular, Manzour et al. (2021) establish that all existing big-$M$ formulations achieve the same continuous relaxation objective function under a mild condition (see Proposition 4). To circumvent this issue, we present a mixed-integer second-order cone program (MISOCP), which gives a tighter continuous relaxation than existing big-$M$ formulations under certain conditions discussed in detail in Section 5.3. This formulation can be solved by powerful state-of-the-art optimization packages. Our numerical results show the superior performance of MISOCP compared to the existing big-$M$ formulations in terms of improving the lower bound and reducing the optimality gap. We also compare our method against the state-of-the-art benchmarks (Chen et al., 2019; Ghoshal and Honorio, 2018) both for identifiable and non-identifiable instances, and show that our method provides the best estimation among all methods in most of the networks, especially for the non-identifiable cases.

The rest of the paper is organized as follows. In Section 2, we define the DAG structure learning problem corresponding to linear SEMs, and give a general framework for the problem. In Section 3, we present our early stopping criterion and establish the asymptotic properties of the solution obtained under this stopping rule. We review existing mathematical formulations in Section 4, and present our proposed mathematical formulations in Section 5. Results of comprehensive numerical studies are presented in Section 6. We end the paper with a summary in Section 7.

2. Problem setup: Penalized DAG estimation with linear SEMs

Let $ℳ=(V,E)$ be an undirected and possibly cyclic super-structure graph with node set $V=\{1,2,\dots ,m\}$ and edge set $E\subseteq V\times V$; let $\overrightarrow{ℳ}=(V,\overrightarrow{E})$ be the corresponding bi-directional graph with $\overrightarrow{E}=\left\{\right(j,k),(k,j)\mid (j,k)\in E\}$. We refer to undirected edges as edges and directed edges as arcs.

We assume that causal effects of continuous random variables in a DAG ${𝒢}_0$ are represented by $m$ linear regressions of the form

$$X_k=\sum _{j\in p{a}_k^{{𝒢}_0}}{\beta }_{jk}{X}_j+{ϵ}_k,k=1,\dots ,m,$$ (1)

where $X_k$ is the random variable associated with node $k,p{a}_k^{{𝒢}_0}$ represents the parents of node $k$ in ${𝒢}_0$, i.e., the set of nodes with arcs pointing to $k$; the latent random variable ${ϵ}_k$ denotes the unexplained variation in node $k$; and BN parameter ${\beta }_{jk}$ specifies the effect of node $j$ on $k$ for $j\in p{a}_k^{{𝒢}_0}$. The above model is known as a linear SEM (Pearl, 2009).

Let $𝒳=\left({𝒳}_1,\dots ,{𝒳}_m\right)$ be the $n\times m$ data matrix with $n$ rows representing i.i.d. samples from each random variable, and $m$ columns representing random variables $X_1,\dots ,X_m$. The linear SEM (1) can be compactly written in matrix form as $𝒳=𝒳B+ℰ$, where $B=\left[\beta \right]\in {\mathbb{R}}^{m\times m}$ is a matrix with ${\beta }_{kk}=0$ for $k=1,\dots ,m,{\beta }_{jk}=0$ for all $(j,k)\notin E$, and $ℰ$ is the $n\times m$ ‘noise’ matrix. Then, $𝒢\left(B\right)$ denotes the directed graph on $m$ nodes such that arc $(j,k)$ appears in $𝒢\left(B\right)$ if and only if ${\beta }_{jk}\ne 0$. Throughout the paper, we will use $B$ and $\beta$ to denote the matrix of coefficients and its vectorized version.

A key challenge when estimating DAGs by minimizing the loss function is that the true DAG is generally not identifiable from observational data. However, for certain SEM distributions, the true DAG is identifiable from observational data; that is when the true causal graph can be identified from the joint distribution. Two important examples are linear SEMs with possibly non-Gaussian homoscedastic noise variables (Peters and Bühlmann, 2013), as well as linear SEMs with unequal noise variances that are known up to a constant (Loh and Bühlmann, 2014). In these special cases, the true DAG can be identified from observational data, without requiring the (strong) ‘faithfulness’ assumption, which is known to be restrictive in high dimensions (Uhler et al., 2013; Sondhi and Shojaie, 2019). Given these important implications, in this paper we focus on learning Bayesian networks corresponding to the above identifiable linear SEMs, i.e., settings where the error variances are either equal, or known up to a constant.

The negative log likelihood for an identifiable linear SEM (1) with equal noise variances is proportional to

$$l\left(\beta ;𝒳\right)=n \mathrm{t}\mathrm{r}\left\{(I-B)(I-B{)}^{\top }\widehat{\Sigma }\right\};$$ (2)

here $\widehat{\Sigma }=n^{-1}{𝒳}^{\top }𝒳$ is the empirical covariance matrix, and $I$ is the identity matrix (Shojaie and Michailidis, 2010; van de Geer and Bühlmann, 2013).

To learn sparse DAGs, van de Geer and Bühlmann (2013) propose to augment the negative log likelihood with an ${\ell }_0$ regularization term. Given a super-structure $ℳ$, the optimization problem corresponding to this penalized negative log-likelihood (PNL$ℳ$) is given by

$$\text{PNL}ℳ\underset{B\in {ℝ}^{m\times m}}{\text{min}}ℒ\left(\beta \right)≔l\left(\beta ;𝒳\right)+{\lambda }_n\Vert \beta {\Vert }_0$$ (3a)

$$\text{s.t. }𝒢\left(B\right)\text{ induces a DAG from }\overrightarrow{ℳ}\text{,}$$ (3b)

where the tuning parameter ${\lambda }_n$ controls the degree of the ${\ell }_0$ regularization

$$\Vert \beta {\Vert }_0≔\sum _{\left(j,k\right)\in \overrightarrow{E}}\mathbb{1}\left({\beta }_{jk}\right),$$

where $\mathbb{1}\left({\beta }_{jk}\right)$ is an indicator function with value one if ${\beta }_{jk}\ne 0$, and 0 otherwise. The constraint (3b) stipulates that the resulting directed subgraph is a DAG induced from $\overrightarrow{ℳ}$. When $ℳ$ corresponds to a complete graph, PNL$ℳ$ reduces to the original PNL of van de Geer and Bühlmann (2013).

The choice of ${\ell }_0$ regularization in (3) is deliberate. Although ${\ell }_1$ regularization has attractive computational and statistical properties in high-dimensional regression (Bühlmann and van de Geer, 2011), many of these advantages disappear in the context of DAG structure learning (Fu and Zhou, 2013; Aragam and Zhou, 2015). By considering ${\ell }_0$ regularization, van de Geer and Bühlmann (2013) establish the consistency of PNL under appropriate assumptions. More specifically, for a Gaussian SEM, they show that the estimated DAG has (asymptotically) the same number of edges as the DAG with minimal number of edges (minimal-edge I-MAP), and establish the consistency of PNL for learning sparse DAGs. These results are formally stated in Proposition 1 in the next section.

Remark 1 A Tikhonov (${\ell }_2$) regularization term, $\mu \Vert \beta {\Vert }_2^2$, for a given $\mu >0$, can also be added to the objective (3a) to obtain more stable solutions (Bertsimas et al., 2016).

In our earlier work (Manzour et al., 2021), we observe that existing mathematical formulations are slow to converge to a provably optimal solution, ${\beta }^{\star }$, of (3) using the state-of-the-art optimization solvers. Therefore, the solution process needs to be terminated early to yield a feasible solution, $\hat{\beta }$ with a positive optimality gap, i.e., a positive difference between the upper bound on $ℒ\left({\beta }^{\star }\right)$ provided by $ℒ\left(\hat{\beta }\right)$ and a lower bound on $ℒ\left({\beta }^{\star }\right)$ provided by the best continuous relaxation obtained by the branch-and-bound algorithm upon termination. However, statistical properties of such a sub-optimal solution are not well-understood. Therefore, there exists a gap between theory and computation: while the optimal solution has nice statistical properties, the properties of the solutions obtained from approximate computational algorithms are not known. Moreover, due to the non-convex and complex nature of the problem, characterizing the properties of the solutions provided by heuristics is especially challenging. In the next section, we bridge this gap by developing a concrete early stopping criterion and establishing the consistency of the solution obtained using this criterion.

3. Early stopping criterion for DAG learning

In this section, we establish a sufficient condition for the approximate solution of PNL$ℳ,\hat{\beta }$ to be consistent for the true coefficients, ${\beta }^0$; that is ${∥{\beta }^0-\hat{\beta }∥}_2^2=𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$, where $s^0$ is the number of arcs in the true DAG, and $x\asymp y$ means that $x$ converges to $y$ asymptotically. This result is obtained by leveraging an important property of the branchand-bound process for integer programming that provides both lower and upper bounds on the objective function $ℒ\left({\beta }^{\star }\right)$ upon early stopping, as well as the consistency results of the PNL estimate with ${\ell }_0$ regularization. Using the insight from this new result, we then propose a concrete stopping criterion for terminating the branch-and-bound process that results in consistent parameter estimates.

Let LB and UB, respectively, denote the lower and upper bounds on the optimal objective function value (3a) obtained from solving (3) under an early stopping criterion (i.e., when the obtained solution is not necessarily optimal). We define the difference between the upper and lower bounds as the absolute optimality gap: GAP = UB – LB. Let $\widehat{𝒢}$ and $\hat{\beta }$ denote the structure of the DAG and coefficients of the arcs from optimization model (3) under the early stopping condition with sample size $n$ and regularization parameter ${\lambda }_n$. Let ${𝒢}^{\star }$ and ${\beta }^{\star }$ denote the DAG structure and coefficients of arcs obtained from the optimal solution of (3), and ${𝒢}^0$ and ${\beta }^0$ denote the true DAG structure and the coefficient of arcs, respectively. We denote the number of arcs in $\widehat{𝒢},{𝒢}^0$, and ${𝒢}^{\star }$ by $\hat{s},s^0$, and $s^{\star }$, respectively. The score value in (3a) of each solution is denoted by $ℒ\left(\varphi \right)$ where $\varphi \in \left\{{\beta }^{\star },\hat{\beta },{\beta }^0\right\}$.

Next, we present our main result. Our result extends van de Geer and Bühlmann’s result on consistency of PNL$ℳ$ for the optimal, but computationally unattainable, estimator, ${\beta }^{\star }$ to an approximate estimator, $\hat{\beta }$, obtained from early stopping. We begin by stating the key result from van de Geer and Bühlmann (2013) and the required assumptions. Throughout, we consider a Gaussian linear SEM of the form (1). We denote the variance of error terms, ${ϵ}_j$, by ${\sigma }_{jj}^2$ and the true covariance matrix of the set of random variables, $\left(X_1,\dots ,X_m\right)$ by the $m\times m$ matrix $\Sigma$.

Assumption 1 Suppose $m<c_{0}n/\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ for some constant $c_0>0$, and for some constant ${\sigma }_0^2{\mathrm{m}\mathrm{a}\mathrm{x}}_{j=1,\dots ,m}{\sigma }_{jj}^2\le {\sigma }_0^2$. Moreover, the smallest and largest eigenvalues of $\Sigma ,{\kappa }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(\Sigma \right)$ and ${\kappa }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(\Sigma \right)$, satisfy

$${\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}<\underset{\_}{\kappa }\le {\kappa }_{\text{min}}\left(\Sigma \right)<{\kappa }_{\text{max}}\left(\Sigma \right)\le \overline{\kappa }<\infty$$

for constants $\underset{\_}{k}$ and $\bar{\kappa }$.

Assumption 2 Let, as in van de Geer and Bühlmann (2013), $\tilde{\Omega }\left(\pi \right)$ be the precision matrix of the vector of noise variables for an SEM given permutation $\pi$ of nodes. Denoting the diagonal entries of this matrix by ${\tilde{\omega }}_{jj}$, there exists a constant ${\omega }_0>0$ such that if $\tilde{\Omega }\left(\pi \right)$ is not a multiple of the identity matrix, then

$$m^{-1}\sum _{j=1}^m{\left({\left({\tilde{\omega }}_{jj}\right)}^2-1\right)}^2>1/{\omega }_0.$$

Proposition 1 (Theorem 5.1 in van de Geer and Bühlmann (2013)) Suppose Assumptions 1 and 2 hold and let ${\alpha }_0≔\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{4}{m},0.05\right\}$. Then for an ${\ell }_0$ regularization parameter $\lambda \asymp \mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n$, it holds with probability at least $1-{\alpha }_0$ that

$${\Vert {\beta }^{\star }-{\beta }^0\Vert }_2^2+\lambda s^{\star }=𝒪\left(\lambda s^0\right).$$

Here, $\lambda ={\lambda }_n/n$, because the loss function (2) is that of van de Geer and Bühlmann (2013) scaled by the sample size $n$. The next result establishes the consistency of the approximate estimator, $\hat{\beta }$, obtained using our proposed early stopping strategy.

Proposition 2 Suppose Assumptions 1 and 2 hold and let ${\alpha }_0=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{4}{m},0.05\right\}$ and $\lambda \asymp \mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n$. Then, the estimator $\hat{\beta }$ obtained from early stopping of the branch-and-bound process such that $\mathrm{G}\mathrm{A}\mathrm{P}\asymp 𝒪\left(n\lambda s^0\right)=𝒪\left(\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)s^0\right)$ satisfies

$${\Vert \hat{\beta }-{\beta }^0\Vert }_2^2\asymp 𝒪\left(\frac{\text{log}\left(m\right)}{n}{s}^0\right)$$

with probability $\left(1-{\alpha }_0\right)$.

Proof First, by the triangle inequality and the fact that $2ab\le a^2+b^2,\forall a,b\in \mathbb{R}$,

$$\begin{gathered} {\Vert \hat{\beta }-{\beta }^0\Vert }_2^2\le {\left({\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2+{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2\right)}^2 \\ ={\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2^2+{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2^2+2{\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2 \\ \le 2{\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2^2+2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2^2. \end{gathered}$$ (4)

Recall that $\beta$ denotes the vectorized coefficient matrix $B$. Then, in a slight abuse of notation, we denote by $𝒳$ both the vectorized and a block diagonal version of $𝒳$ and by $ℰ$ a vectorized version of error $ℰ$. Then $\ell (\beta ;𝒳)$ can be written as $\ell (\beta ;𝒳)=\Vert 𝒳-𝒳\beta {\Vert }_2^2$ (see Eq. 1). Then, we can write a Taylor series expansion of $\ell (\hat{\beta };𝒳)$ around $\ell \left({\beta }^{\star };𝒳\right)$ to get

$${\Vert 𝒳\left(\hat{\beta }-{\beta }^{\star }\right)\Vert }_2^2=\ell \left(\hat{\beta };𝒳\right)-\ell \left({\beta }^{\star };𝒳\right)-2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }𝒳\left({\beta }^{\star }-{\beta }^0\right)+2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }ℰ.$$ (5)

But, Proposition 2.1 in Vershynin (2012) states that for every $0<\xi <1$,

$$\Vert \widehat{\Sigma }-\Sigma {\Vert }_2\le c_{\xi }{\left(\frac{m}{n}\right)}^{1/2},$$ (6)

with probability $1-\xi$, where $c_{\xi }$ is a constant depending only on $\xi$. Letting $\xi ={\alpha }_0$, (6) holds in our setting with probability $1-{\alpha }_0$ and constant $c_{{\alpha }_0}$.

Since, by Assumption 1, ${\kappa }_{\text{min}}\left(\Sigma \right)\ge \underset{\_}{\kappa }>\frac{c_0}{\text{log}\left(n\right)}$, by a corollary of Weyl’s theorem (Wainwright, 2019, Ch. 6, Eq. 6.7), ${\mathrm{m}\mathrm{a}\mathrm{x}}_j \left|{\kappa }_j\left(\widehat{\Sigma }\right)-{\kappa }_j\left(\Sigma \right)\right|\le \Vert \widehat{\Sigma }-\Sigma {\Vert }_2$, where ${\kappa }_j\left(\Sigma \right)$ denotes the $j$th eigenvalue of $\Sigma$. Using the fact that by Assumption 1, $m\le c_{0}n/\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ for a suitable constant $c_0$, we have

$${\kappa }_{\text{min}}\left({𝒳}^{\top }𝒳\right)=n{\kappa }_{\text{min}}\left(\widehat{\Sigma }\right)>n\underset{\_}{\kappa }-{(mn)}^{1/2}>n\underset{\_}{\kappa }-n{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}=n\left(\underset{\_}{\kappa }-{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\right),$$

which means that ${\kappa }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left({𝒳}^{\top }𝒳\right)>0$ with probability $1-{\alpha }_0$.

Denoting $c_n^{\prime }\equiv {\left(\underset{\_}{\kappa }-{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\right)}^{-1}$, for large enough $n$, we have that, with probability $1-{\alpha }_0$,

$${\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2^2\le n^{-1}{c}_n^{\prime }{\Vert 𝒳\left(\hat{\beta }-{\beta }^{\star }\right)\Vert }_2^2.$$ (7)

Adding $n^{-1}{c}_n^{\prime }\lambda \hat{s}$ (which is non-negative) to the right-hand-side of (7), combining it with (5), and using triangle inequality again, we get

$$\begin{gathered} \Vert \hat{\beta }-{\beta }^{\star }\Vert { }_2^2\le n^{-1}{c}_n^{\prime }{\Vert 𝒳\left(\hat{\beta }-{\beta }^{\star }\right)\Vert }_2^2+n^{-1}{c}_n^{\prime }\lambda \hat{s} \\ =n^{-1}{c}_n^{\prime }\left(\ell \left(\hat{\beta };𝒳\right)-\ell \left({\beta }^{\star };𝒳\right)-2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }𝒳\left({\beta }^{\star }-{\beta }^0\right)+2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }ℰ\right)+n^{-1}{c}_n^{\prime }\lambda \hat{s} \\ \le n^{-1}{c}_n^{\prime }\left|\ell \left(\hat{\beta };𝒳\right)-\ell \left({\beta }^{\star };𝒳\right)+\lambda \hat{s}+\left(\lambda s^{\star }-\lambda s^{\star }\right)-2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }𝒳\left({\beta }^{\star }-{\beta }^0\right)+2{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }ℰ\right| \\ \le n^{-1}{c}_n^{\prime }\left|\ell \left(\hat{\beta };𝒳\right)-\ell \left({\beta }^{\star };𝒳\right)+\lambda \hat{s}-\lambda s^{\star }\right|+n^{-1}{c}_n^{\prime }\lambda s^{\star }+2n^{-1}{c}_n^{\prime }{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }𝒳\left({\beta }^{\star }-{\beta }^0\right)+2n^{-1}{c}_n^{\prime }\left|{\left(\hat{\beta }-{\beta }^{\star }\right)}^{\top }{𝒳}^{\top }ℰ\right| \\ \le n^{-1}{c}_n^{\prime }\left|ℒ\left(\hat{\beta };𝒳\right)-ℒ\left({\beta }^{\star };𝒳\right)\right|+n^{-1}{c}_n^{\prime }\lambda s^{\star }+2n^{-1}{c}_n^{\prime }{\kappa }_{\text{max}}\left({𝒳}^{\top }𝒳\right){\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2+2n^{-1}{c}_n^{\prime }{\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2{\Vert {𝒳}^{\top }ℰ\Vert }_2, \end{gathered}$$ (8)

where, as before, ${\kappa }_{\mathrm{m}\mathrm{a}\mathrm{x}}$ denotes the maximum eigenvalue of the matrix.

Using a similar argument as the one used above for the minimum eigenvalue of ${𝒳}^{\top }𝒳$, by (6) we have that, with probability $1-{\alpha }_0$,

$${\kappa }_{\text{max}}\left({𝒳}^{\top }𝒳\right)=n{\kappa }_{\text{max}}\left(\widehat{\Sigma }\right)\le n{\kappa }_{\text{max}}\left(\Sigma \right)+n{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\le n\left(\overline{\kappa }+{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\right).$$

Plugging the above bound into (8) we get

$${\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2^2\le n^{-1}{c}_n^{\prime }\left|ℒ\left(\hat{\beta };𝒳\right)-ℒ\left({\beta }^{\star };𝒳\right)\right|+n^{-1}{c}_n^{\prime }\lambda s^{\star }+2c_n^{\prime }\left(\overline{\kappa }+{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\right){\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2+2n^{-1}{c}_n^{\prime }{\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2{\Vert {𝒳}^{\top }ℰ\Vert }_2.$$ (9)

Now, let $Z={\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2,\Pi =2c_n^{\prime }\left[\left(\overline{k}+{\left(\frac{c_0}{\text{log}\left(n\right)}\right)}^{1/2}\right){\Vert {\beta }^{\star }-{\beta }^0\Vert }_2+n^{-1}{\Vert {𝒳}^{\top }ℰ\Vert }_2\right]$, and $\Gamma =n^{-1}{c}_n^{\prime }\left|ℒ(\hat{\beta };𝒳)-ℒ\left({\beta }^{\star };𝒳\right)\right|$. Then, the inequality in (9) can be written as $Z^2\le \Pi Z+\Gamma$. Solving for $Z$ and noting that $Z,\Gamma$ and $\Pi$ are non-negative, in order to have $Z^2\le \Pi Z+\Gamma$, we must have $Z\le \left(\Pi +\sqrt{{\Pi }^2+4\Gamma }\right)/2$.

Next, let $𝒯$ be the event under which $\Pi =o\left(1\right)$. Then, on this set, we have $Z\le \left(o\right(1)+\sqrt{o\left(1\right)+4\Gamma })/2$, or, $Z^2\le \Gamma +o\left(1\right)$; that is

$${\Vert \hat{\beta }-{\beta }^{\star }\Vert }_2^2\le n^{-1}{c}_n^{\prime }\left|ℒ\left(\hat{\beta };𝒳\right)-ℒ\left({\beta }^{\star };𝒳\right)\right|+n^{-1}{c}_n^{\prime }\lambda s^{\star }+o\left(1\right).$$ (10)

Plugging (10) into (4), on the set $𝒯$ we have

$${\Vert \hat{\beta }-{\beta }^0\Vert }_2^2\le 2n^{-1}{c}_n^{\prime }\left|ℒ\left(\hat{\beta };𝒳\right)-ℒ\left({\beta }^{\star };𝒳\right)\right|+2n^{-1}{c}_n^{\prime }\lambda s^{\star }+2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2^2+o\left(1\right).$$ (11)

Or, using the fact that $ℒ\left(\hat{\beta }\right)-ℒ\left({\beta }^{\star }\right)\le \mathrm{G}\mathrm{A}\mathrm{P}$,

$${\Vert \hat{\beta }-{\beta }^0\Vert }_2^2\le 2n^{-1}{c}_n^{\prime }\text{GAP}+2{\Vert {\beta }^{\star }-{\beta }^0\Vert }_2^2+2n^{-1}{c}_n^{\prime }\lambda s^{\star }+o\left(1\right).$$ (12)

Now, by Proposition 1, we know that with probability at least $1-{\alpha }_0,{∥{\beta }^{\star }-{\beta }^0∥}_2^2=𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$, and $\lambda s^{\star }=𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$. Moreover, using the arguments in van de Geer and Bühlmann (2013), for $n^{-1}{∥{𝒳}^{\top }ℰ∥}_2$ (bounds for set ${𝒯}_1$ in Section 7.4.1 of that paper), the probability of the set $𝒯$ is lower bounded by the probability that ${∥{\beta }^{\star }-{\beta }^0∥}_2^2=𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$, which is $1-{\alpha }_0$. Thus, if we stop the branch-and-bound algorithm when

$$\text{GAP}=𝒪\left(n\lambda s^0\right)=𝒪\left(\text{log}\left(m\right)s^0\right),$$

then the first two terms in (12) would both be of order $𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$, while the third term, $2n^{-1}{c}_n^{\prime }\lambda s^{\star }$ would be of a smaller order (by an $n^{-1}$ factor). This guarantees that, with probability at least $\left(1-{\alpha }_0\right),{∥\hat{\beta }-{\beta }^0∥}_2^2=𝒪\left(s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n\right)$, as desired.■

Proposition 2 suggests that the branch-and-bound algorithm can be stopped by setting a threshold $c^{\star }n\lambda s^0$ on the value of $\mathrm{G}\mathrm{A}\mathrm{P}=|\mathrm{U}\mathrm{B}-\mathrm{L}\mathrm{B}|$ for a constant $c^{\star }>0$, say $c^{\star }=1$. Such a solution will then achieve the same desirable statistical properties (in terms of parameter consistency) as the optimal solution ${\beta }^{\star }$. While $\lambda$ can be chosen data-adaptively (as discussed in Section 6), both of these choices depend on the value of $s^0$, which is not known in practice. However, one can find an upper bound for $s^0$ based on the number of edges in the super-structure $ℳ$. In particular, if $ℳ$ is the moral graph (Pearl, 2009) with $s_m$ edges, then $s^0\le s_m$.

However, while in many applications $s_m\asymp s^0$, this is not always guaranteed. Thus, to ensure consistent estimation when replacing $s^0$ with $s_m$ and setting $c^{\star }=1$ in practice, we use the more conservative threshold of $\lambda s^0\asymp s^0\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)/n$. With this choice the first and third terms in (12) would be of the same (vanishing) order, and the consistency rate would be driven by the convergence rate of ${∥{\beta }^{\star }-{\beta }^0∥}_2^2$. We investigate the performance of this choice in Section 6.4.

The above results, including the specific choice of early stopping criterion, are also valid if the super-structure $ℳ$ corresponding to the moral graph is not known a priori. That is because the moral graph can be consistently estimated from data using recent developments in graphical modeling; see Drton and Maathuis (2017) for a review of the literature. While some of the existing algorithms based on ${\ell }_1$-penalty require an additional irrepresentability condition (Meinshausen and Bühlmann, 2006; Saegusa and Shojaie, 2016), this assumption can be relaxed by using instead an adaptive lasso penalty or by thresholding the initial lasso estimates (Bühlmann and van de Geer, 2011).

In light of Proposition 2, it is of great interest to develop algorithms that converge to a solution with a small optimality gap expeditiously. To achieve this, one approach is to obtain better lower bounds using the branch-and-bound process from strong mathematical formulations for (3). To this end, we next review existing formulations of (3).

4. Existing Formulations of DAG Learning with Linear SEMs

In this section, we provide a brief review of known mathematical formulations for DAG learning with linear SEMs and refer the reader to Manzour et al. (2021) for more detailed descriptions. We first outline the necessary notation below.

Index Sets

$V=\{\mathrm{1,2},\dots ,m\}$: index set of random variables;

$𝒟=\{\mathrm{1,2},\dots ,n\}$: index set of samples.

Input

$ℳ=(V,E)$: an undirected super-structure graph (e.g., the moral graph);

$\overrightarrow{ℳ}=(V,\overrightarrow{E})$: the bi-directional graph corresponding to the undirected graph $ℳ$;

$𝒳=\left({𝒳}_1,\dots ,{𝒳}_m\right)$, where ${𝒳}_v={\left(x_{1v},x_{2v},\dots ,x_{nv}\right)}^{\top }$ and $x_{dv}$ denotes $d$th sample $(d\in 𝒟)$ of random variable $X_v$; note $𝒳\in {\mathbb{R}}^{n\times m};$

${\lambda }_n$: tuning parameter (penalty coefficient for ${\ell }_0$ regularization).

Continuous optimization variables

${\beta }_{jk}$: weight of arc $(j,k)$ representing the regression coefficients $\forall (j,k)\in \overrightarrow{E}$.

Binary optimization variables

$g_{jk}=1$ if ${\beta }_{jk}\ne 0$; otherwise 0, $\forall (j,k)\in \overrightarrow{E}$.

Let $F(\beta ,g)={\sum }_{k\in V} {\sum }_{d\in 𝒟} {\left(x_{dk}-{\sum }_{(j,k)\in \overrightarrow{E}} {\beta }_{jk}{x}_{dj}\right)}^2+{\lambda }_n{\sum }_{(j,k)\in \overrightarrow{E}} g_{jk}$. The PNL$ℳ$ problem can be cast as the following optimization model:

$$\underset{B\in {ℝ}^{m\times m},g\in \left\{0,1\right\}\left|\overrightarrow{E}\right|}{\text{min}}F\left(\beta ,g\right),$$ (13a)

$$𝒢\left(B\right)\text{ induces a DAG from }\overrightarrow{ℳ},$$ (13b)

$${\beta }_{jk}\left(1-g_{jk}\right)=0,\forall \left(j,k\right)\in \overrightarrow{E},$$ (13c)

$$g_{jk}\in \left\{0,1\right\},\forall \left(j,k\right)\in \overrightarrow{E}.$$ (13d)

The objective function (13a) is an expanded version of $ℒ\left(\beta \right)$ in PNL$ℳ$, where we use the indicator variable $g_{jk}$ to encode the ${\ell }_0$ regularization. The constraints in (13b) rule out cycles. The constraints in (13c) are non-linear and stipulate that ${\beta }_{jk}\ne 0$ only if $g_{jk}=1$.

There are two sources of difficulty in solving (13a)–(13d): (i) the acyclic nature of DAG imposed by the combinatorial constraints in (13b); (ii) the set of nonlinear constraints in (13c), which stipulates that ${\beta }_{jk}\ne 0$ only if there exists an arc $(j,k)$ in $𝒢\left(B\right)$. In Section 4.1, we discuss related studies to address the former, whereas in Section 4.2 we present relevant literature for the latter.

4.1. Linear encodings of the acyclicity constraints (13b)

There are several ways to ensure that the estimated graph does not contain any cycles. The first approach is to add a constraint for each cycle in the graph, so that at least one arc in this cycle must not exist in $𝒢\left(B\right)$. A cutting plane (CP) method is used to solve such a formulation which may require generating an exponential number of constraints. Another way to rule out cycles is by imposing constraints such that the nodes follow a topological order (Park and Klabjan, 2017). A topological ordering is a unique ordering of the nodes of a graph from 1 to $m$ such that the graph contains an arc $(j,k)$ if node $j$ appears before node $k$ in the order. We refer to this formulation as topological ordering (TO). The TO formulation has $𝒪\left(m^2\right)$ variables and $𝒪\left(\right|\overrightarrow{E}\left|\right)$ constraints. We give these formulations in the Appendix, for completeness.

The layered network (LN) formulation for learning from continuous data proposed by Manzour et al. (2021) is shown to perform better, empirically, than the TO formulation because it reduces the number of binary variables and is proven to obtain the same continuous relaxation bounds. Therefore, smaller quadratic programs are solved that provide the same relaxation bounds as larger quadratic programs. This formulation is closely related to the generation number approach proposed in Cussens (2010). A layered network is a network whose nodes can be assigned to layers with associated layer values such that there exists no arc from nodes in layer $v$ to nodes in other layers $u<v$. In this paper, we focus on the LN formulation and refer the reader to the Appendix and Manzour et al. (2021) for comparisons of these formulations and their sizes in detail. Next, we give the LN encoding of the acyclicity constraints (see, also Cussens, 2010). Define decision variables $g_{jk}\in \left\{\mathrm{0,1}\right\}$ for all $(j,k)\in \overrightarrow{E}$, where, as before, the variable $g_{jk}$ takes value 1 if ${\beta }_{jk}\ne 0$, and

$$LN1-m+m{g}_{jk}\le {\psi }_k-{\psi }_j\forall \left(j,k\right)\in \overrightarrow{E}.$$ (14a)

Here ${\psi }_k$ is the layer value for node $k$, where $1\le {\psi }_k\le m$. The set of constraints in (14a) ensures that if there exists an arc $(j,k)$ in the DAG, then layer of node $j$ should be before that of node $k$, i.e., ${\psi }_k\ge {\psi }_j+1$. This rules out any cycles. Furthermore, binary vector $g$ helps correctly encode the ${\ell }_0$ regularization. The LN formulation has $𝒪\left(\right|\overrightarrow{E}\left|\right)$ variables and constraints. Note that $\left|\overrightarrow{E}\right|$ is much smaller than $m^2$ for sparse skeleton/moral graphs.

4.2. Linear encodings of the non-convex constraints (13c)

The nonconvexity of the set of constraints in (13c) causes challenges in obtaining provably optimal solutions with existing optimization software. Therefore, we consider convex representations of this set of constraints. First, we present a linear encoding of the constraints in (13c). Although the existing compact (i.e., polynomial sized) TO and LN formulations discussed in Section 4.1 differ in their approach to ruling out cycles, one commonality among them is that they replace the non-linear constraint (13c) by so called big-M constraints given by

$$-M{g}_{jk}\le {\beta }_{jk}\le M{g}_{jk},\forall \left(j,k\right)\in \overrightarrow{E},$$ (15)

for a large enough $M$. Unfortunately, these big-$M$ constraints (15) are poor approximations of (13c), especially in this problem, because no natural and tight value for $M$ exists. Although a few techniques have been proposed for obtaining the big-$M$ parameter for sparse regression problem (e.g., Bertsimas et al., 2016; Bertsimas and Van Parys, 2020; Gómez and Prokopyev, 2021; Park and Klabjan, 2020), the resulting parameters are often too large in practice. Further, finding a tight big-$M$ parameter itself is a difficult problem to solve for DAG structure learning.

Consider (13a)–(13d) by replacing (13c) with the linear big-$M$ constraints (15) and writing the objective function in a matrix form. We denote the resulting formulation, which has a convex quadratic objective and linear constraints, by the following MIQP.

$$\mathbf{\text{MIQP}}\underset{B\in {ℝ}^{m\times m},g\in \left\{0,1\right\}\left|\overrightarrow{E}\right|}{\text{min}}\text{\hspace{0.17em}\hspace{0.17em}tr}\left[\left(I-B\right){(I-B)}^{\top }{𝒳}^{\top }𝒳\right]+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk}$$ (16a)

$$\left(13\mathrm{b}\right),\left(15\right)$$ (16b)

$$g_{jk}\in \left\{\mathrm{0,1}\right\}\forall \left(j,k\right)\in \overrightarrow{E}.$$ (16c)

Depending on which types of constraints are used in lieu of (13b), as explained in Section 4.1, MIQP (16) results in three different formulations: MIQP+CP, which uses (23), MIQP+TO, which uses (24), and MIQP+LN, which uses (14), respectively.

To discuss the challenges of the big-$M$ approach, we give a definition followed by two propositions.

Definition 2 A formulation $A$ is said to be stronger than formulation $B$ if $ℛ\left(A\right)\subset ℛ\left(B\right)$ where $ℛ\left(A\right)$ and $ℛ\left(B\right)$ correspond to the feasible regions of continuous relaxations of $A$ and $B$, respectively.

Proposition 3 (Proposition 3 in Manzour et al. (2021)) The MIQP+TO and MIQP+CP formulations are stronger than the MIQP+LN formulation.

As a consequence of Definition 2, the optimal objective function value of the continuous relaxation of the stronger formulation provides a lower bound on the true optimal objective function of the MIQP that is greater than or equal to the optimal objective function value of the continuous relaxation of the weaker formulation due to the smaller set of feasible solutions. However, the next proposition shows that, perhaps surprisingly, the continuous relaxations of MIQP+TO and MIQP+CP formulations, while stronger according to Definition 2, give the same optimal objective function value (and the same lower bound on the true optimal objective).

Proposition 4 (Proposition 5 in Manzour et al. (2021)) Let ${\beta }_{jk}^{\star }$ denote the optimal coefficient associated with an arc $(j,k)\in \overrightarrow{E}$ from problem (3). For the same variable branching in the branch-and-bound process, the continuous relaxations of the MIQP+LN formulation for ${\ell }_0$ regularization attain the same optimal objective function value as MIQP+TO and MIQP+CP, if $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^{\star }\right|$.

Proposition 3 implies that the MIQP+TO and MIQP+CP formulations are stronger than the MIQP+LN formulation. Nonetheless, Proposition 4 establishes that for sufficiently large values of $M$, stronger formulations for the DAG learning problem attain the same continuous relaxation objective function value as the weaker formulation throughout the branch-and-bound tree. The optimal solution to the continuous relaxation of MIQP formulations of DAG structure learning may not be at an extreme point of the convex hull of feasible points. Hence, stronger formulations do not necessarily ensure better lower bounds for certain formulations of this problem involving the nonlinear objective. This is in contrast to a mixed-integer program (MIP) with linear objective, whose continuous relaxation is a linear program (LP). In that case, there exists an optimal solution that is an extreme point of the corresponding feasible set. As a result, a better lower bound can be obtained from a stronger formulation that better approximates the convex hull of the set of solutions to a mixed-integer linear program; this generally leads to faster convergence. A prime example is the traveling salesman problem (TSP), for which stronger formulations attain better computational performance (Öncan et al., 2009). In contrast, the numerical results by Manzour et al. (2021) empirically show that MIQP+LN has better computational performance because it is a compact formulation with the fewest constraints and the same continuous relaxation bounds.

Our next result, which is adapted from Dong et al. (2015) to the DAG structure learning problem, shows that the continuous relaxation of MIQP is equivalent to the optimal solution to the problem where the ${\ell }_0$-regularization term is replaced with an ${\ell }_1$-regularization term (i.e., $\left.\Vert \beta {\Vert }_1={\sum }_{(j,k)\in \overrightarrow{E}} \left|{\beta }_{jk}\right|\right)$ with a particular choice of the ${\ell }_1$ penalty. This motivates us to consider tighter continuous relaxation for MIQP. Let $\left({\beta }^R,g^R\right)$ be an optimal solution to the continuous relaxation of MIQP.

Proposition 5 For $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$, a continuous relaxation of MIQP (16), where the binary variables are relaxed, is equivalent to the problem where the ${\ell }_0$ regularization term is replaced with an ${\ell }_1$-regularization term with penalty parameter $\tilde{\lambda }=\frac{{\lambda }_n}{M}$.

Proof For $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$, the value $g_{jk}^R$ is $\frac{{\beta }_{jk}^R}{M}$ in an optimal solution to the continuous relaxation of MIQP (16). Otherwise, we can reduce the value of the decision variable $g^R$ without violating any constraints while reducing the objective function. Note that since $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$, we have $\frac{{\beta }_{jk}^R}{M}\le 1,\forall (j,k)\in \overrightarrow{E}$. To show that the set of constraints in (13b) is satisfied, we consider the set of CP constraints. In this case, the set of constraints (13b) holds, i.e., ${\sum }_{(j,k)\in {𝒞}_A} \frac{{\beta }_{jk}^R}{M}\le \left|{𝒞}_A\right|-1,\forall {𝒞}_A\in 𝒞$, because $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$. This implies that $g_{jk}^R=\frac{{\beta }_{jk}^R}{M}$ is the optimal solution. Thus, the objective function reduces to ${\ell }_1$ regularization with the coefficient $\frac{{\lambda }_n}{M}$.

Finally, Proposition 4 establishes that for $M\ge 2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^{\star }\right|$, the objective function value of the continuous relaxations of MIQP+CP, MIQP+LN and MIQP+TO are equivalent. This implies that the continuous relaxations of all formulations are equivalent, which completes the proof.■

Despite the promising performance of MIQP+LN, its continuous relaxation objective function value provides a weak lower bound due to the big-$M$ constraints. To circumvent this issue, a natural strategy is to improve the big-$M$ value. Nonetheless, existing methods which ensure a valid big-$M$ value or heuristic techniques (Park and Klabjan, 2017; Gómez and Prokopyev, 2021) do not lead to tight big-$M$ values. For instance, the heuristic technique by Park and Klabjan (2017) to obtain big-$M$ values always satisfies the condition in Proposition 5 and exact techniques are expected to produce even larger big-$M$ values. Therefore, we directly develop tighter approximations for (13c) next.

5. New Perspectives for Mathematical Formulations of DAG Learning

In this section, we discuss improved mathematical formulations for learning DAG structure of a BN based on convex (instead of linear) encodings of the constraints in (13c).

Problem (13) is an MIQP with non-convex complementarity constraints (13c), a class of problems which has received a fair amount of attention from the operations research community over the last decade (Frangioni and Gentile, 2006, 2007, 2009; Frangioni et al., 2011; Gómez and Prokopyev, 2021; Liu et al., 2023; Wei et al., 2023, 2022). There has also been recent interest in leveraging these developments to solve sparse regression problems with ${\ell }_0$ regularization (Pilanci et al., 2015; Dong et al., 2015; Xie and Deng, 2020; Atamtürk and Gómez, 2019; Wei et al., 2020).

Next, we review applications of MIQPs with complementarity constraints of the form (13c) for solving sparse regression with ${\ell }_0$ regularization. Frangioni et al. (2011) develop a so-called projected perspective relaxation method, to solve the perspective relaxation of mixed-integer nonlinear programming problems with a convex objective function and complementarity constraints. This reformulation requires that the corresponding binary variables are not involved in other constraints. Therefore, it is suitable for ${\ell }_0$ sparse regression, but cannot be applied for DAG structure learning. Pilanci et al. (2015) show how a broad class of ${\ell }_0$-regularized problems, including sparse regression as a special case, can be formulated exactly as optimization problems. The authors use the Tikhonov regularization term $\mu \Vert \beta {\Vert }_2^2$ and convex analysis to construct an improved convex relaxation using the reverse Huber penalty. In a similar vein, Bertsimas and Van Parys (2020) exploit the Tikhonov regularization and develop an efficient algorithm by reformulating the sparse regression mathematical formulation as a saddle-point optimization problem with an outer linear integer optimization problem and an inner dual quadratic optimization problem which is capable of solving high-dimensional sparse regressions. Xie and Deng (2020) apply the perspective formulation of sparse regression optimization problem with both ${\ell }_0$ and the Tikhonov regularizations. The authors establish that the continuous relaxation of the perspective formulation is equivalent to the continuous relaxation of the formulation given by Bertsimas and Van Parys (2020). Dong et al. (2015) propose perspective relaxation for ${\ell }_0$ sparse regression optimization formulation and establish that the popular sparsity-inducing concave penalty function known as the minimax concave penalty (Zhang, 2010) and the reverse Huber penalty (Pilanci et al., 2015) can be obtained as special cases of the perspective relaxation – thus the relaxations of formulations by Zhang (2010); Pilanci et al. (2015); Bertsimas and Van Parys (2020); Xie and Deng (2020) are equivalent. The authors obtain an optimal perspective relaxation that is no weaker than any perspective relaxation. Among the related approaches, the optimal perspective relaxation by Dong et al. (2015) is the only one that does not explicitly require the use of Tikhonov regularization.

The perspective formulation, which in essence is a fractional non-linear program, can be cast either as a mixed-integer second-order cone program (MISOCP) or a semi-infinite mixed-integer linear program (SIMILP). Both formulations can be solved directly by state-of-the-art optimization packages. Dong et al. (2015) and Atamtürk and Gómez (2019) solve the continuous relaxations and then use a heuristic approach (e.g., rounding techniques) to obtain a feasible solution (an upper bound). In this paper, we directly solve the MISOCP and SIMILP formulations for learning sparse DAG structures.

Next, we present how the perspective formulation can be suitably applied for DAG structure learning with ${\ell }_0$ regularization. We further cast the problem as MISOCP and SIMILP. To this end, we express the objective function (16a) in the following way:

$$\begin{gathered} \text{tr}\left[\left(I-B\right){(I-B)}^{\top }{𝒳}^{\top }𝒳\right]+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk} \\ =\text{tr}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+2B{B}^{\top }{𝒳}^{\top }𝒳\right]+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk}. \end{gathered}$$ (17)

Let $\delta \in {\mathbb{R}}_{+}^m$ be a vector such that ${𝒳}^{\top }𝒳-D_{\delta }⪰0$, where $D_{\delta }=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\delta }_1,\dots ,{\delta }_m\right)$ and $A⪰0$ means that matrix $A$ is positive semi-definite. By splitting the quadratic term ${𝒳}^{\top }𝒳=\left({𝒳}^{\top }𝒳-D_{\delta }\right)+D_{\delta }$ in (17), the objective function can be expressed as

$$\text{tr}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+B{B}^{\top }\left({𝒳}^{\top }𝒳-D_{\delta }\right)\right]+\text{tr}\left(B{B}^{\top }{D}_{\delta }\right)+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk}.$$ (18)

Let $Q={𝒳}^{\top }𝒳-D_{\delta }$. (In the presence of Tikhonov regularization with tuning parameter $\mu >0$, we let $Q={𝒳}^{\top }𝒳+\mu I-D_{\delta }$ as described in Remark 1.) Then, Cholesky decomposition can be applied to decompose $Q$ as $q^{\top }q$ (note $\left.Q⪰0\right)$. As a result, $\mathrm{t}\mathrm{r}\left(B{B}^{\top }Q\right)=\mathrm{t}\mathrm{r}\left(B{B}^{\top }{q}^{\top }q\right)={\sum }_{i=1}^m {\sum }_{j=1}^m {\left({\sum }_{(\ell ,j)\in \overrightarrow{E}} {\beta }_{\ell j}{q}_{i\ell }\right)}^2$. The separable component can also be expressed as $\mathrm{t}\mathrm{r}\left(B{B}^{\top }{D}_{\delta }\right)={\sum }_{j=1}^m {\sum }_{(j,k)\in \overrightarrow{E}} {\delta }_j{\beta }_{jk}^2$. Using this notation, the objective (18) can be written as

$$\mathrm{t}\mathrm{r}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+B{B}^{\top }Q\right]+\sum _{j=1}^m \sum _{(j,k)\in \overrightarrow{E}} {\delta }_j{\beta }_{jk}^2+{\lambda }_n\sum _{(j,k)\in \overrightarrow{E}} g_{jk}.$$

The Perspective Reformulation (PRef) of MIQP is then given by

$$\mathbf{\text{PRef }}\underset{B\in {ℝ}^{m\times m},g\in \left\{0,1\right\}\left|\overrightarrow{E}\right|}{\text{min}}\text{\hspace{0.17em}\hspace{0.17em}tr}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+B{B}^{\top }Q\right]+\sum _{j=1}^m\sum _{\left(j,k\right)\in \overrightarrow{E}}{\delta }_j\frac{{\beta }_{jk}^2}{g_{jk}}+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk},$$ (19a)

$$\left(16\mathrm{b}\right)-\left(16\mathrm{c}\right)\text{.}$$ (19b)

The objective function (19a) is formally undefined when some $g_{jk}=0$. More precisely, we use the convention that $\frac{{\beta }_{jk}^2}{g_{jk}}=0$ when ${\beta }_{jk}=g_{jk}=0$ and $\frac{{\beta }_{jk}^2}{g_{jk}}=+\infty$ when ${\beta }_{jk}\ne 0$ and $g_{jk}=0$ (Frangioni and Gentile, 2009). The continuous relaxation of PRef, referred to as the perspective relaxation, is much stronger than the continuous relaxation of MIQP under certain conditions discussed in detail in Section 5.3 (Pilanci et al., 2015). However, an issue with PRef is that the objective function is nonlinear due to the fractional term. There are two ways to reformulate PRef. One as a mixed-integer second-order conic program (MISOCP) (see, Section 5.1) and the other as a semi-infinite mixed-integer linear program (SIMILP) (see, Section 5.2).

5.1. Mixed-integer second-order conic program

Let $s_{jk}$ be additional variables representing ${\beta }_{jk}^2$. Then, the MISOCP formulation is given by

$$\text{MISOCP}\underset{B\in {ℝ}^{m\times m},s\in {ℝ}^{\left|\overrightarrow{E}\right|}\left|,g\in \left\{0,1\right\}\right|\overrightarrow{E}\mid }{\text{min}}\text{\hspace{0.17em}\hspace{0.17em}tr}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+B{B}^{\top }Q\right]+\sum _{j=1}^m\sum _{\left(j,k\right)\in \overrightarrow{E}}{\delta }_j{s}_{jk}+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk},$$ (20a)

$$s_{jk}{g}_{jk}\ge {\beta }_{jk}^2(j,k)\in \overrightarrow{E},$$ (20b)

$$0\le s_{jk}\le M^2{g}_{jk}(j,k)\in \overrightarrow{E},$$ (20c)

$$\left(16\mathrm{b}\right)-\left(16\mathrm{c}\right).$$ (20d)

Here, the constraints in (20b) imply that ${\beta }_{jk}\ne 0$ only when $g_{jk}=1$. The constraints in (20b) are second-order conic representable because they can be written in the form of $\sqrt{4{\beta }_{jk}^2+{\left(s_{jk}-g_{jk}\right)}^2}\le s_{jk}+g_{jk}$. The set of constraints in (20c) is valid since ${\beta }_{jk}\le M{g}_{jk}$ implies ${\beta }_{jk}^2\le M^2{g}_{jk}^2=M^2{g}_{jk}^2$ and $g_{jk}^2=g_{jk}$ for $g_{jk}\in \left\{\mathrm{0,1}\right\}$. The set of constraints in (20c) is not required, yet they improve the computational efficiency especially when we restrict the big-$M$ value. Xie and Deng (2020) report similar behavior for sparse regression. When we relax $g_{jk}\in \left\{\mathrm{0,1}\right\}$ and let $g_{jk}\in \left[\mathrm{0,1}\right]$, we obtain the continuous relaxation of MISOCP (20). Let us denote the feasible region of continuous relaxation of MISOCP (20) and MIQP (16) by $ℛ\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{O}\mathrm{C}\mathrm{P}$ and $ℛ\mathrm{M}\mathrm{I}\mathrm{Q}\mathrm{P}$, and the objective function values by $\mathrm{O}\mathrm{F}\mathrm{V}\left(ℛ\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{O}\mathrm{C}\mathrm{P}\right)$ and $\mathrm{O}\mathrm{F}\mathrm{V}\left(ℛ\mathrm{M}\mathrm{I}\mathrm{Q}\mathrm{P}\right)$, respectively. For a more general problem than ours, Cui et al. (2013) give a detailed proof establishing that the feasible region of the former is contained in the feasible region of latter i.e., $ℛ\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{O}\mathrm{C}\mathrm{P}\subset ℛ\mathit{M}\mathit{I}\mathit{Q}\mathit{P}$, and $\mathrm{O}\mathrm{F}\mathrm{V}\left(ℛ\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{O}\mathrm{C}\mathrm{P}\right)\ge \mathrm{O}\mathrm{F}\mathrm{V}\left(ℛ\mathrm{M}\mathrm{I}\mathrm{Q}\mathrm{P}\right)$. Therefore, we are able to obtain stronger lower bounds using MISOCP than MIQP under suitable choices for $D_{\delta }$ as described in Section 5.3.

5.2. Mixed-integer semi-infinite integer linear program

An alternative approach to reformulate PRef is via perspective cuts developed by Frangioni and Gentile (2006, 2007). To apply perspective cuts, we use the reformulation idea first proposed in Frangioni and Gentile (2006) by introducing dummy decision matrix $D$ to distinguish the separable and non-separable part of the objective function; we also add the additional constraint $d=\beta$ where $d_{jk}$ is $(j,k)$ element of matrix $D$ and $\beta$ is the decision variable in the optimization problem. Following this approach, MIQP can be reformulated as an SIMILP:

$$\text{SIMILP}\underset{B\in {ℝ}^{m\times m},v\in {ℝ}^{\mid \overrightarrow{E}\mid },g\in \{0,1{\}}^{\mid \overrightarrow{E}\mid }}{\text{min}}\text{\hspace{0.17em}\hspace{0.17em}tr}\left[\left(I-B-B^{\top }\right){𝒳}^{\top }𝒳+D{D}^{\top }Q\right]+ \sum _{j=1}^m\sum _{\left(j,k\right)\in \overrightarrow{E}}{\delta }_j{v}_{jk}+{\lambda }_n\sum _{\left(j,k\right)\in \overrightarrow{E}}{g}_{jk},$$ (21a)

$$d_{jk}={\beta }_{jk}\left(j,k\right)\in \overrightarrow{E},$$ (21b)

$$v_{jk}\ge 2{\bar{\beta }}_{jk}{\beta }_{jk}-{\bar{\beta }}_{jk}^2{g}_{jk}\forall {\bar{\beta }}_{jk}\in [-M,M]\forall (j,k)\in \overrightarrow{E},$$ (21c)

$$\left(16\mathrm{b}\right)-\left(16\mathrm{c}\right),$$ (21d)

$$v_{jk}\ge 0,\left(j,k\right)\in \overrightarrow{E}.$$ (21e)

The set of constraints in (21c) is known as perspective cuts. Note that there are infinitely many such constraints. Although this problem cannot be solved directly, it lends itself to a delayed cut generation approach whereby a (small) finite subset of constraints in (21c) is kept, the current solution $\left({\beta }^{\star },g^{\star },v^{\star }\right)$ of the relaxation is obtained, and all the violated inequalities for the relaxation solution are added for ${\bar{\beta }}_{jk}=\frac{{\beta }_{jk}^{\star }}{g_{jk}^{\star }}$ (assuming $\frac{0}{0}=0$). This process is repeated until termination criteria are met. This procedure can be implemented using the cut callback function available by off-the-shelf solvers such as Gurobi or CPLEX.

5.3. Selecting $\delta$

In the MISOCP and SIMILP formulations, one important question is how to identify a valid $\delta$. A natural choice is $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(\delta \right)={\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}e$, where ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}$ is the minimum eigenvalue of ${𝒳}^{\top }𝒳$ and $e$ is a column vector of ones. The issue with this approach is that if ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}=0$, then $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(\delta \right)$ becomes a trivial 0 matrix. If $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(\delta \right)$ turns out to be a zero matrix, then MISOCP formulation reduces to the big-$M$ formulation. Frangioni and Gentile (2007) present an effective approach for obtaining a valid $\delta$ by solving the following semidefinite program (SDP)

$$\underset{\delta \in {ℝ}^{\left|V\right|}}{\text{max}}\left\{\sum _{i\in V}{\delta }_i:{𝒳}^{\top }𝒳-\text{diag}\left(\delta \right)\underset{\_}{\succ }0,{\delta }_i\ge 0\right\}.$$ (22a)

This formulation can attain a non-zero $D_{\delta }$ even if ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}=0$. Numerical results by Frangioni and Gentile (2007) show that this method compares favorably with the minimum eigenvalue approach. Zheng et al. (2014) propose an SDP approach, which obtains $D_{\delta }$ such that the continuous relaxation of MISOCP (20) is as tight as possible.

Similar to Dong et al. (2015), our formulation does not require adding a Tikhonov regularization. In this case, PRef is effective when ${𝒳}^{\top }𝒳$ is sufficiently diagonally dominant. When $n\ge m$ and each row of $𝒳$ is independent, then ${𝒳}^{\top }𝒳$ is guaranteed to be a positive semi-definite matrix (Dong et al., 2015). On the other hand, when $n<m,{𝒳}^{\top }𝒳$ is not full-rank. Therefore, a Tikhonov regularization term should be added with sufficiently large $\mu$ to make ${𝒳}^{\top }𝒳+\mu I⪰0$ (Dong et al., 2015) in order to benefit from the strengthening provided by PRef.

6. Experiments

In this section, we report the results of our numerical experiments that compare different formulations and evaluate the effect of different tuning parameters and estimation strategies. Our experiments are performed on a cluster operating on UNIX with Intel Xeon E5–2640v4 2.4GHz. All formulations are implemented in the Python programming language. Gurobi 8.1 is used as the solver. Unless otherwise stated, a time limit of 50m (in seconds), where $m$ denotes the number of nodes, and an MIQP relative optimality gap of 0.01 are imposed across all experiments after which runs are aborted. The relative optimality gap is calculated by $\mathrm{R}\mathrm{G}\mathrm{A}\mathrm{P}≔\frac{UB\left(X\right)-LB\left(X\right)}{UB\left(X\right)}$ where UB(X) denotes the objective value associated with the best feasible integer solution (incumbent) and LB(X) represents the best obtained lower bound during the branch-and-bound process for the formulation $X\in \{\mathrm{M}\mathrm{I}\mathrm{Q}\mathrm{P},\mathrm{S}\mathrm{I}\mathrm{M}\mathrm{I}\mathrm{L}\mathrm{P},\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{O}\mathrm{C}\mathrm{P}\}$.

Unless otherwise stated, we assume ${\lambda }_n=\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ which corresponds to the Bayesian information criterion (BIC) score. To select the big-$M$ parameter, $M$, in all formulations we use the proposal of Park and Klabjan (2017). Specifically, given ${\lambda }_n$, we solve each problem without cycle prevention constraints and obtain ${\beta }^R$. We then use the upper bound $M=2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$. Although this value does not guarantee an upper bound for $M$, the results provided in Park and Klabjan (2017) and Manzour et al. (2021) computationally confirm that this approach gives a large enough value of $M$.

The goals of our computational study are twofold. First, we compare the various mathematical formulations to determine which gives us the best performance in Subsection 6.1, compare the sensitivity to the model parameters in Subsection 6.2, and the choice of the regularization term in Subsection 6.3. Second, in Subsection 6.4 we use the best-performing formulation to investigate the implications of the early stopping condition on the quality of the solution with respect to the true graph. To be able to perform such a study, we use synthetic data so that the true graph is available. In Subsection 6.5, we compare our algorithm against two state-of-the-art benchmark algorithms on publicly available datasets.

We use the package pcalg in R to generate random graphs. First, we create a DAG by randomDAG function and assign random arc weights (i.e., $\beta$) from a uniform distribution, $𝒰\left[\mathrm{0.1,1}\right]$. Next, the resulting DAG and random coefficients are fed into the rmvDAG function to generate multivariate data based on linear SEMs (columns of matrix $\chi$) with the standard normal error distribution. We consider $m\in \{10,20,30,40\}$ nodes and $n=100$ samples. The average outgoing degree of each node, denoted by $d$, is set to two. We generate 10 random Erdős-Rényi graphs for each setting $(m,n,d)$. We observe that in our instances, the minimum eigenvalue of ${𝒳}^{\top }𝒳$ across all instances is 3.26 and the maximum eigenvalue is 14.21.

Two types of problem instances are considered: (i) a set of instances with known moral graph corresponding to the true DAG; (ii) a set of instances with a complete undirected graph, i.e., assuming no prior knowledge. We refer to the first class of instances as moral instances and to the second class as complete instances. The observational data, $𝒳$, for both classes of instances are the same. The function moralize(graph) in the pcalg R-package is used to generated the moral graph from the true DAG. Although the moral graph can be consistently estimated from data using penalized estimation procedures with polynomial complexity (e.g., Loh and Bühlmann, 2014), the quality of moral graph affects all optimization models. Therefore, we use the true moral graph in our experiments, unless otherwise stated.

6.1. Comparison of Mathematical Formulations

We use the following MIQP-based metrics to measure the quality of a solution: relative optimality gap (RGAP), computation time in seconds (Time), Upper Bound (UB), Lower Bound (LB), objective function value (OFV) of the initial continuous relaxation, and the number of explored nodes in the branch-and-bound tree (# BB). An in-depth analysis comparing the existing mathematical formulations that rely on linear encodings of the constraints in (13c) for MIQP formulations is conducted by Manzour et al. (2021). The authors conclude that the MIQP+LN formulation outperforms the other MIQP formulations, and the promising performance of MIQP+LN can be attributed to its size: (1) MIQP+LN has fewer binary variables and constraints than MIQP+TO, (2) MIQP+LN is a compact (polynomial-sized) formulation in contrast to MIQP+CP which has an exponential number of constraints. Therefore, in this paper, we analyze the formulations based on the convex encodings of the constraints in (13c).

6.1.1. Comparison of MISOCP formulations

We next experiment with MISOCP formulations. For the set of constraints in (13b), we use LN, TO, and CP constraints discussed in Section 4.1 resulting in three formulations denoted as MISOCP+LN, MISOCP+TO, MISOCP+CP, respectively. The MISOCP+TO formulation fails to find a feasible solution for instances with 30 and 40 nodes, see Table 1. For moral instances, the optimality gaps for MISOCP+TO are 0.000 and 0.021 for instances with 10 and 20 nodes, respectively; for complete instances, the optimality gap for MISOCP+TO formulation are 0.009 and 0.272 for instances with 10 and 20 nodes, respectively. Moreover, Table 1 illustrates that MISOCP+LN performs better than MISOCP+TO for even small instances (i.e., 10 and 20 nodes).

Table 1:

Optimality gaps for MISOCP+TO and MISOCP+LN formulations

	Moral		Complete
m	MISOCP+TO	MISOCP+LN	MISOCP+TO	MISOCP+LN
10	0.000	0.000	0.009	0.008
20	0.021	0.006	0.272	0.195
30	-	0.010	-	0.195
40	-	0.042	-	0.436

“-” denotes that no feasible solution, i.e., UB, is obtained within the time limit, so optimality gap cannot be computed.

For MISOCP+CP, instead of incorporating all constraints given by (23), we begin with no constraint of type (23). Given an integer solution with cycles, we detect a cycle and impose a new cycle prevention constraint to remove the detected cycle. Depth First Search (DFS) can detect a cycle in a directed graph with complexity $O\left(\right|V|+|E\left|\right)$. Gurobi LazyCallback function is used, which allows adding cycle prevention constraints in the branch-and-bound algorithm, whenever an integer solution with cycles is found. The same approach is used by Park and Klabjan (2017) to solve the corresponding MIQP+CP. Note that Gurobi solver follows a branch-and-cut implementation and adds many general-purpose and special-purpose cutting planes.

Figures 1a and 1b show that MISOCP+LN outperforms MISOCP+CP in terms of relative optimality gap and computational time. In addition, MISOCP+LN attains better upper and lower bounds than MISOCP+CP (see, Figures 1c and 1d). MISOCP+CP requires the solution of a second-order cone program (SOCP) after each cut, which reduces its computational efficiency and results in higher optimality gaps than MISOCP+LN. MISOCP+TO requires many binary variables which makes the problem very inefficient when the network becomes denser and larger as shown in Table 1. Therefore, we do not illustrate the MISOCP+TO results in Figure 1.

Figure 1:

Optimization-based measures for MISOCP+LN (left bar) and MISOCP+CP (right bar) formulations for $n=100$.

6.1.2. Comparison of MISOCP versus SIMILP

Our computational experiments show that the SIMILP formulation generally performs poorly when compared to MISOCP+LN and MIQP+LN in terms of optimality gap, upper bound, and computational time. We report the results for SIMILP+LN, MISOCP+LN, and MIQP+LN formulations in Figure 2. We only consider the LN formulation because that is the best performing model among the alternatives both for MISOCP and MIQP formulations.

Figure 2:

Optimization-based measures for MISOCP+LN, MIQP+LN, and SIMILP+LN formulations for $n=100$.

Figures 2a and 2b show the relative optimality gaps and computational times for these three formulations. Figures 2c and 2d demonstrate that SIMILP+LN attains lower bounds that are comparable with other two formulations. In particular, for complete instances with large number of nodes, SIMILP+LN attains better lower bounds than MIQP+LN. Nonetheless, SIMILP+LN fails to obtain good upper bounds. Therefore, the relative optimality gap is considerably larger for SIMILP+LN.

The poor performance of SIMILP+LN might be because state-of-the-art optimization packages (e.g., Gurobi, CPLEX) use many heuristics to obtain a good feasible solution (i.e., upper bound) for a compact formulation. In contrast, SIMILP is not a compact formulation, and we build the SIMILP gradually by adding violated constraints iteratively. Therefore, a feasible solution to the original formulation is not available while solving the relaxations with a subset of the constraints (21c). Moreover, the optimization solvers capable of solving MISOCP formulations have witnessed noticeable improvement due to theoretical developments in this field. In particular, Gurobi reports 20% and 38% improvement in solution time for versions 8 and 8.1, respectively. In addition, Gurobi v8.1 reports over four times faster solution times than CPLEX for solving MISOCP on their benchmark instances.

6.1.3. Comparison of MISOCP versus MIQP formulations

In this section, we demonstrate the benefit of using the second-order conic formulation MISOCP+LN instead of the linear big-$M$ formulation MIQP+LN. As before, we only consider the LN formulation for this purpose. Figures 3a and 3b show that MISOCP+LN performs better than MIQP+LN in terms of the average relative optimality gap across all number of nodes $m\in \{10,20,30,40\}$. The only exception is $m=40$ for moral instances, for which MIQP+LN performs better than MISOCP+LN. Nonetheless, we observe that MISOCP+LN clearly outperforms MIQP+LN for complete instances which are more difficult to solve.

Figure 3:

Optimization-based measures for MISOCP+LN, MIQP+LN formulations for $n=100$.

Figures 3c and 3d show the performance of both formulations in terms of the resulting upper and lower bounds on the objective function. We observe that MISOCP+LN attains better lower bounds especially for complete instances. However, MISOCP+LN cannot always obtain a better upper bound. In other words, MISOCP+LN is more effective in improving the lower bound instead of the upper bound as expected.

Figures 3e and 3f show that MISOCP+LN uses fewer branch-and-bound nodes and achieves better continuous relaxation values than MIQP+LN.

6.2. Analyzing the Choices of ${\lambda }_n$ and $M$

We now experiment on different values for ${\lambda }_n$ and $M$ to assess the effects of these parameters on the performance of MISOCP+LN and MIQP+LN. First, we consider multiple $\lambda$ values, ${\lambda }_n\in \{\mathrm{l}\mathrm{o}\mathrm{g}(n),2\mathrm{l}\mathrm{o}\mathrm{g}(n),4\mathrm{l}\mathrm{o}\mathrm{g}(n\left)\right\}$, while keeping the value of $M$ the same (i.e., $M=2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^{\star }\right|$). Table 2 shows that as ${\lambda }_n$ increases, MISOCP+LN consistently performs better than MIQP+LN in terms of the relative optimality gap, computational time, the number of branch-and-bound nodes, and continuous relaxation objective function value. Indeed, the difference becomes even more pronounced for more difficult cases (i.e., complete instances). For instance, for ${\lambda }_n=4\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)=18.4$, the relative optimality gap reduces from 0.465 to 0.374, an over 24% improvement. In addition, MISOCP+LN allows more instances to be solved to optimality within the time limit. For example, for moral instances with $m=40,{\lambda }_n=18.4$, eight out of ten instances are solved to optimality using MISOCP+LN while only two instances are solved to optimality by MIQP+LN.

Table 2:

Computational results for different values of ${\lambda }_n=t\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ for $t\in \{\mathrm{1,2},4\}$, * indicates that the problem is solved to the optimality tolerance. Superscript ${ }^i$ indicates that out of ten runs, $i$ instances finish before hitting the time limit. Time is averaged over instances that solve within the time limit, RGAP is averaged over instances that reach the time limit. Better RGAPs are in bold.

Moral										Complete
Instances		RGAP		Time		# nodes		Relaxation OFV		RGAP		Time		# nodes		Relaxation OFV
$m$	${\lambda }_n$	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP
10	4.6	*	*	3	2	1306	3715	738.7	664.9	*	*	65	74	38850	114433	724.4	629.3
10	9.2	*	*	4	3	1116	2936	784.6	693.5	*	*	31	39	15736	55543	772.5	662.2
10	18.4	*	*	3	2	1269	2457	857.0	747.5	*	*	26	29	18223	41197	844.5	720.2
20	4.6	*	*	69	51	46513	76261	1474.2	1325.8	.195	.275	1000	1000	101509	238765	1404.9	1144.5
20	9.2	*	*	26	27	10695	31458	1589.6	1406.8	.152	.250	1000	1000	152206	274514	1526.9	1238.6
20	18.4	*	*	24	36	9574	33788	1763.7	1552.7	.113 2	.208	944	1000	159789	277687	1697.1	1395.0
30	4.6	.010 8	0.0118	378	527	121358	514979	2230.1	2037.7	.298	.441	1500	1500	38474	64240	2024.0	1569.7
30	9.2	*	*	104	291	33371	248190	2392.4	2168.5	.239	.395	1500	1500	59034	71475	2217.5	1741.5
30	18.4	*	*	48	74	15649	57909	2608.3	2383.8	.215	.318	1500	1500	74952	96586	2449.2	2006.9
40	4.6	.0426	.037 4	1551	1615	664496	2565247	2979.3	2748.6	.436	.545	2000	2000	23083	49050	2582.0	1946.3
40	9.2	.024 8	.0364	1125	1336	353256	1347702	3200.7	2923.5	.397	.473	2000	2000	29279	73917	2869.9	2216.9
40	18.4	.024 8	.0352	1099	1375	434648	1137666	3521.8	3225.4	.374	.465	2000	2000	31298	60697	3240.1	2633.1

Finally, we study the influence of the big-$M$ parameter. Instead of a coefficient $\gamma =2$ in Park and Klabjan (2017), we experiment with $M=\gamma \underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$ for $\gamma \in \{2,5,10\}$ in Table 3, where $\left|{\beta }_{jk}^R\right|$ denotes the optimal solution of each optimization problem without the constraints to remove cycles. The larger the big-$M$ parameter, the worse the effectiveness of both models. However, comparing the continuous relaxation objective function values, we observe that MISOCP+LN tightens the formulation using the conic constraints whereas MIQP+LN does not have any means to tighten the formulation instead of big-$M$ constraints which have poor relaxation. In most cases, the MISOCP+LN formulation allows more instances to be solved to optimality than MIQP+LN. For larger $m$, because Gurobi solves larger SOCP relaxations in each branch-and-bound node, the MISOCP+LN formulation explores far fewer branch-and-bound nodes and stops with a similar RGAP at termination. For $M>2\underset{\left(j,k\right)\in \overrightarrow{E}}{\text{max}}\left|{\beta }_{jk}^R\right|$, MISOCP+LN outperforms MIQP+LN in all measures, in most cases.

Table 3:

Computational results for different values of $\gamma$, * indicates that the problem is solved to the optimality tolerance. Superscript ${ }^i$ indicates that out of ten runs, $i$ instances finish before hitting the time limit. Time is averaged over instances that solve within the time limit, RGAP is averaged over instances that reach the time limit. Better RGAPs are in bold.

Moral										Complete
Instances		RGAP		Time		# nodes		Relaxation OFV		RGAP		Time		# nodes		Relaxation OFV
$m$	$\gamma$	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP
10	2	*	*	3	2	1306	3715	738.7	664.9	*	*	65	74	38850	114433	724.4	629.3
10	5	*	*	5	2	1433	3026	717.9	647.1	*	*	81	82	42675	130112	705.1	607.8
10	10	*	*	5	2	1523	2564	712.5	641.1	*	*	74	100	35576	174085	699.8	600.3
20	2	*	*	69	51	46513	76261	1474.2	1325.8	.195	.275	1000	1000	101509	238765	1404.9	1144.5
20	5	*	*	103	156	65951	209595	1438.2	1274.2	.211	.308	1000	1000	97940	225050	1375.3	1080.9
20	10	*	*	215	207	150250	349335	1427.7	1256.6	.230	.310	1000	1000	90864	257998	1366.3	1058.2
30	2	.010 8	.0118	378	527	121358	514979	2230.1	2037.7	.298	.441	1500	1500	38474	64240	2024.0	1569.7
30	5	.011 8	.0148	571	620	164852	527847	2173.9	1950.3	.336	.474	1501	1500	33120	64339	1969.4	1448.4
30	10	.0248	.014 8	630	638	202635	585234	2156.5	1919.6	.349	.480	1500	1500	30579	77100	1951.2	1404.0
40	2	.0426	.037 4	1551	1615	664496	2565247	2979.3	2748.6	.436	.545	2000	2000	23083	49050	2582.0	1946.3
40	5	.045 6	.0472	1643	1634	638323	1347868	2895.6	2635.0	.579	.580	2000	2000	12076	30858	2488.0	1751.7
40	10	.056 4	.0572	1639	1632	599281	1584187	2869.2	2595.6	.585	.594	2000	2000	11847	30222	2456.1	1679.6

6.3. The Effect of Tikhonov Regularization

In this subsection, we consider the effect of adding a Tikhonov regularization term to the objective (see Remark 1) by considering $\mu \in \{0,\mathrm{l}\mathrm{o}\mathrm{g}(n),2\mathrm{l}\mathrm{o}\mathrm{g}(n\left)\right\}$ while keeping the values of ${\lambda }_n=\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ and $M$ the same as before. Table 4 demonstrates that for all instances with $\mu >0$, MISOCP+LN outperforms MIQP+LN. For complete instances with $m=40$ and $\mu =9.2$, MISOCP+LN improves the optimality gap from 0.445 to 0.367, an improvement over 21%. The reason for this improvement is that $\mu >0$ makes the matrix more diagonally dominant; therefore, it makes the conic constraints more effective in tightening the formulation and obtaining a better optimality gap. Also, MISOCP+LN allows more instances to be solved to optimality than MIQP+LN.

Table 4:

Computational results for different values of $\mu$, * indicates that the problem is solved to the optimality tolerance. Superscript ${ }^i$ indicates that out of ten runs, $i$ instances finish before hitting the time limit. Time is averaged over instances that solve within the time limit, RGAP is averaged over instances that reach the time limit. Better RGAPs are in bold.

Moral										Complete
Instances		RGAP		Time		# nodes		Relaxation OFV		RGAP		Time		# nodes		Relaxation OFV
m	$\mathit{\mu }$	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP	MISOCP	MIQP
10	0	*	*	3	2	1306	3715	738.7	664.9	*	*	65	74	38850	114433	724.4	629.3
10	4.6	*	*	4	2	1043	2758	802.0	708.5	*	*	69	72	38778	119825	789.3	675.7
10	9.2	*	*	4	2	1067	2231	858.0	748.1	*	*	72	74	36326	114383	843.2	712.3
20	0	*	*	69	51	46513	76261	1474.2	1325.8	.195	.275	1000	1000	101509	238765	1404.9	1144.5
20	4.6	*	*	45	45	15111	55302	1604.1	1426.5	.167	.242	1000	1000	102467	249490	1551.7	1267.1
20	9.2	*	*	43	55	15384	62297	1716.8	1515.7	.142	.223	1000	1000	94360	258194	1668.3	1355.1
30	0	.010 8	.0118	378	527	121358	514979	2230.1	2037.7	.298	.441	1500	1500	38474	64240	2024.0	1569.7
30	4.6	.008 9	.0118	310	392	76668	358544	2432.5	2187.7	.237	.387	1500	1500	45473	69258	2286.4	1788.5
30	9.2	.009 9	.0108	67	377	12410	320632	2612.6	2311.4	.209	.367	1500	1500	41241	68661	2484.3	1915.7
40	0	.0426	.037 4	1551	1615	664496	2565247	2979.3	2748.6	.436	.545	2000	2000	23083	49050	2582.0	1946.3
40	4.6	.027 8	.0294	1331	1620	422654	1303301	3281.6	2972.8	.354	.471	2000	2000	13209	30995	2985.4	2261.3
40	9.2	.020 8	.0286	870	1507	239214	1762210	3575.4	3165.3	.367	.445	2000	2000	13884	54638	3321.7	2468.7

6.4. Practical Implications of Early Stopping

In this subsection, we evaluate the quality of the estimated DAGs obtained from MISOCP+LN by comparing them with the ground truth DAG. To this end, we use three measures: the average structural Hamming distance (SHD) which counts the number of arc differences (additions, deletions, or reversals) required to transform the estimated DAG to the true DAG, the average false positive rate (FPR) which is the proportion of edges appearing in the estimated DAG but not the true DAG and the average true positive rate (TPR) which is the proportion of edges appearing in both the true DAG and the estimated DAG. Finally, because the convergence of the branch-and-bound process may be slow in some cases, we set a time limit of 100m.

To test the quality of the solution obtained with an early stopping criterion, we set the absolute optimality gap parameter as $GAP=\frac{\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)}{n}{s}_m$ and the ${\ell }_0$ regularization parameter as ${\lambda }_n=\mathrm{l}\mathrm{o}\mathrm{g}m$ as suggested by the discussion following Proposition 2 for achieving a consistent estimate. We compare the resulting suboptimal solution to the solution obtained by setting GAP = UB – LB = 0 to obtain the truly optimal solution.

Table 5 shows the numerical results for the average solution time (in seconds) for instances that are solved within the time limit, the number of instances that were not solved within the time limit, the actual absolute optimality gap at termination, the average FPR, the average TPR, the average SHD of the resulting DAGs, across 10 runs for moral instances. Table 5 indicates that the average SHD for $GAP=\frac{\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)}{n}{s}_m$ is close to that of the truly optimal solution, and the average FPR and TPR are the same between setting $\mathrm{G}\mathrm{A}\mathrm{P}=\frac{\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)}{n}{s}_m$ and GAP = 0 except for $m=10$. Note that a lower GAP generally leads to a better SHD score. From a computational standpoint, we observe that by using the early stopping criterion, we are able to obtain consistent solutions faster in some scenarios. In particular, for these instances, the average solution time reduces by 26% for $m=30$ and 32% for $m=40$, for the seven and two instances that solve before the time limit, respectively. The number of instances that are solved before hitting the 100m time limit are the same for $GAP=0$ and $\mathrm{G}\mathrm{A}\mathrm{P}=\frac{\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)}{n}{s}_m$. Furthermore, stopping early does not sacrifice too much from the quality of the resulting DAG as can be seen from the SHD scores.

Table 5:

Structural Hamming distances (SHD), False Positive Rate (FPR) and True Positive Rate (TPR) for early stopping with $n=100,{\lambda }_n=\mathrm{l}\mathrm{o}\mathrm{g}\left(m\right)$, $\mathrm{G}\mathrm{A}\mathrm{P}\le \tau$ for moral instances. The superscripts ${ }^i$ indicate that out of ten runs, $i$ instances finish before hitting the time limit. Time is averaged over instances that solve within the time limit, GAP, RGAP, SHD, FPR and TPR are averaged over all instances.

		$\tau =0$						$\tau =\frac{\text{log}\left(m\right)}{n}{s}_m$
$m$	$s_m$	Time	GAP	RGAP	SHD	FPR	TPR	Time	GAP	RGAP	SHD	FPR	TPR
10	19	1.2810	0.00	0.000	0.75	0.04	1.00	1.2810	0.00	0.000	0.77	0.02	1.00
20	58	6.159	0.70	0.000	1.50	0.01	1.00	6.049	1.33	0.001	2.00	0.01	1.00
30	109	37.407	7.75	0.002	1.67	0.00	1.00	27.637	10.59	0.003	1.66	0.00	1.00
40	138	935.002	43.02	0.010	5.00	0.01	1.00	640.152	45.04	0.011	5.00	0.01	1.00

6.5. Comparison to Other Benchmarks

In this section, we compare the performance of MISOCP against the state-of-the-art benchmarks. These experiments are executed on a laptop with a Windows 10 operating system, an Intel Core i7-8750H 2.2-GHz CPU, 8-GB DRAM using Python 3.8 with Gurobi 9.1.1 Optimizer.

The benchmarks considered in this section include the top-down approach (EqVarDAG-TD) and the high-dimensional top-down approach (EqVarDAG-HD-TD) of Chen et al. (2019), as well as the high-dimensional bottom-up approach (EqVarDAG-HD-BU) of Ghoshal and Honorio (2018). By taking advantage of the conditions for identifiability in linear SEM models, these benchmark procedures offer polynomial-time algorithms for learning DAGs by iteratively identifying a source (top-down) or sink (bottom-up) node based on solving a series of covariance selection problems.

We compare the performance of the methods on twelve publicly available networks from Manzour et al. (2021) and Bayesian Network Repository (bnlearn). The number of nodes in these networks ranges from $m=6$ to $m=70$. We generate data from both identifiable and non-identifiable error distributions. In the case of identifiable distributions (ID), we generate the data by using random arc weights $\beta$ from $𝒰[-1,-0.1]\cup 𝒰\left[\mathrm{0.1,\; 1}\right]$ and $n=500$ samples standard normal errors. The data for the non-identifiable (NID) error distributions was generated similarly, but from normal errors with non-equal error variances chosen randomly from {0.5, 1, 1.5}.

As an input superstructure graph to MISOCP, other than the true moral graphs, we also consider a superstructure estimate based on the empirical correlation matrix (CorEst). This estimate—which is guaranteed to be a super set of the DAG skeleton under the faithfulness assumption—was obtained by testing whether each correlation coefficient is nonzero at 0.05 significance level; the p-values were obtained using the Fisher’s Z-transformation for correlation coefficients. The MISOCP with true and correlation matrix superstructures are denoted as MISOCP-True and MISOCP-CorEst, respectively, in Table 6. A time limit of 50m (seconds), $\lambda =2\mathrm{l}\mathrm{o}\mathrm{g}\left(n\right)$ and the Gurobi RGAP of 0.01 are imposed across the experiments.

Table 6:

The comparison between MISOCP and the state-of-the-art EqVarDAG methods of Chen et al. (2019) and Ghoshal and Honorio (2018).

		EqVarDAG-HD-BU					EqVarDAG-HD-TD					EqVarDAG-TD					MISOCP-CorEst						MISOCP-True
Data	$\text{Network}(m)$	Time	SHD	SHDs	TPR	FPR	Time	SHD	SHDs	TPR	FPR	Time	SHD	SHDs	TPR	FPR	Time	RGAP SHD		SHDs	TPR	FPR	Time	RGAP SHD		SHDs	TPR	FPR
	Dsep(6)	≤ 1	3	3	1	.333	≤ 1	3	3	1	.333	≤ 1	0	0	1	0	≤ 1	*	0	0	1	0	≤ 1	*	0	0	1	0
	Asia(8)	≤ 1	0	0	1	0	≤ 1	0	0	1	0	≤ 1	0	0	1	0	≤ 1	*	0	0	1	0	≤ 1	*	0	0	1	0
	Bowling(9)	≤ 1	5	4	1	.2	≤ 1	1	1	1	.04	≤ 1	0	0	1	0	≤ 1	*	0	0	1	0	≤ 1	*	0	0	1	0
	InsSmall(15)	≤ 1	11	8	1	.137	≤ 1	2	2	1	.025	≤ 1	0	0	1	0	≥ 750	.05	0	0	1	0	4	*	0	0	1	0
	Rain(14)	≤ 1	5	4	1	.068	≤ 1	2	2	1	.027	≤ 1	0	0	1	0	155	*	0	0	1	0	1	*	0	0	1	0
ID	Cloud(16)	≤ 1	15	12	1	.149	≤ 1	6	6	1	.059	≤ 1	0	0	1	0	≥ 800	.101	0	0	1	0	≤ 1	*	0	0	1	0
	Puimel(18)	≤ 1	10	9	1	.074	≤ 1	4	4	1	.03	≤ 1	0	0	1	0	5	*	0	0	1	0	≤ 1	*	0	0	1	0
	Galaxy(20)	≤ 1	16	13	1	.095	≤ 1	9	9	1	.054	≤ 1	1	1	1	.006	≥ 1000	.048	0	0	1	0	2	*	0	0	1	0
	Insurance(27)	2	27	23	.962	.084	2	11	11	1	.037	≤ 1	0	0	1	0	≥ 1350	.216	0	0	1	0	176	*	0	0	1	0
	Factors(27)	2	25	25	.971	.081	2	17	17	.985	.057	≤ 1	3	3	.956	0	≥ 1350	.157	0	0	1	0	≥ 1350	.036	0	0	1	0
	Hailfinder(56)	10	78	70	.985	.052	11	19	19	1	.013	≤ 1	1	1	.985	0	≥ 2800	.21	13	12	.955	.007	≥ 2800	.052	0	0	1	0
	Hepar2(70)	17	147	139	.959	.062	19	70	70	.967	.029	2	1	1	.992	0	≥ 3500	.371	134	122	.878	.052	≥ 3500	.096	0	0	1	0
	Dsep(6)	≤ 1	4	3	1	.444	≤ 1	2	1	1	.222	≤ 1	1	1	.833	0	≤ 1	*	1	1	.833	0	≤ 1	*	1	1	.833	0
	Asia(8)	≤ 1	16	13	.875	.75	≤ 1	6	4	.875	.25	≤ 1	6	4	.875	.25	≤ 1	*	4	2	.875	.15	≤ 1	*	6	3	.875	.25
	Bowling(9)	≤ 1	7	6	.909	.24	≤ 1	7	5	.909	.24	≤ 1	4	2	.909	.12	≤ 1	*	2	1	.909	.04	≤ 1	*	4	2	.909	.12
	InsSmall(15)	≤ 1	24	19	.96	.287	≤ 1	8	7	.96	.087	≤ 1	2	1	.96	.012	≥ 750	.029	7	5	.88	.05	12	*	2	1	.96	.012
	Rain(14)	≤ 1	17	12	.944	.219	≤ 1	10	7	.944	.123	≤ 1	4	1	.944	.041	43	*	7	4	.833	.055	2	*	4	1	.944	.041
NID	Cloud(16)	≤ 1	19	14	.895	.168	≤ 1	20	14	.895	.178	≤ 1	12	6	.895	.099	≥ 800	.062	8	5	.947	.069	≤ 1	*	8	2	.947	.069
	Punnel(18)	≤ 1	12	11	.944	.081	≤ 1	6	6	.944	.037	≤ 1	1	1	.944	0	8	*	1	1	.944	0	≤ 1	*	1	1	.944	0
	Galaxy(20)	≤ 1	36	28	.955	.208	≤ 1	27	20	.955	.155	≤ 1	17	10	.955	.095	≥ 1000	.02	8	5	.909	.036	2	*	6	3	.955	.03
	Insurance(27)	2	40	32	1	.134	2	23	19	.981	.074	≤ 1	13	10	.962	.037	≥ 1350	.191	11	9	.923	.023	≥ 1350	.055	6	4	.962	.013
	Factors(27)	2	12	11	.971	.035	2	32	24	.882	.085	≤ 1	32	24	.765	.057	≥ 1350	.111	19	15	.853	.032	≥ 1350	.062	22	16	.868	.046
	Hailfinder(56)	8	103	88	.97	.069	8	90	70	.97	.06	2	59	40	.894	.035	≥ 2800	.156	20	11	.985	.013	≥ 2800	.096	15	8	.985	.009
	Hepar2(70)	14	110	95	.992	.048	16	73	61	.992	.031	2	52	40	.862	.015	≥ 3500	.199	33	23	.935	.011	≥ 3500	.112	36	18	.96	.014

Measures of performance of the benchmark algorithms are summarized in columns EqVarDAG-TD, EqVarDAG-HD-TD, and EqVarDAG-HD-BU of Table 6. The column Time reports the solution time in seconds. For all datasets, the true networks can be used to evaluate the quality of the estimated networks. We report SHD, TPR, and FPR for all the estimated networks. Given that the true causal network cannot be recovered in the setting of non-identifiable data (NID), we also report the structural SHD between the undirected skeleton of the true DAG and the corresponding skeleton of estimated network; this is denoted as SHDs in Table 6.

We observe that most of the EqVarDAG methods solve the problem within a second. With respect to the quality of the estimation, EqVarDAG-TD provides better performance in SHD compared to EqVarDAG-HD-TD and EqVarDAG-HD-BU. The column RGAP reports the relative gap at early termination. The symbol (*) denotes that the problem is solved to the optimality tolerance. Compared with the benchmarks, MISOCP with a CorEst or true superstructure requires longer solution times; however, MISOCP consistently provides high SHD and SHDs scores in every network. Moreover, MISOCP is able to provide the best estimation among all methods in most of the networks.

Finally, we highlight that in the non-identifiable datasets (NID), MISOCP clearly outperforms the benchmarks. This is, perhaps, not surprising, as the benchmark algorithms heavily rely on the identifiability assumption and are not guaranteed to work if this assumption is violated.

7. Conclusion

In this paper, we study the problem of learning an optimal directed acyclic graph (DAG) from continuous observational data, where the causal effect among the random variables is linear. The central problem is a quadratic optimization problem with regularization. We present a mixed-integer second order conic program (MISOCP) which entails a tighter relaxation than existing formulations with linear constraints. Our numerical results show that MISOCP can successfully improve the lower bound and results in better optimality gap when compared with other formulations based on big-$M$ constraints, especially for dense and large instances. Moreover, we establish an early stopping criterion under which we can terminate branch-and-bound and achieve a solution which is asymptotically optimal. In addition, we show that our method outperforms two state-of-the-art algorithms, especially on non-identifiable datasets.

Appendix A. Alternative linear encodings of constraints (13b)

There are several ways to ensure that the estimated graph does not contain any cycles. The first approach is to add a constraint for each cycle in the graph, so that at least one arc in this cycle must not exist in $𝒢\left(B\right)$. A cutting plane (CP) method is used to solve such a formulation which may require generating an exponential number of constraints (Jaakkola et al., 2010). In particular, let $𝒞$ be the set of all possible directed cycles and ${𝒞}_A\in 𝒞$ be the set of arcs defining a cycle. The CP formulation removes cycles by imposing the following constraints for (13b)

$$CP\sum _{\left(j,k\right)\in {𝒞}_A}{g}_{jk}\le \left|{𝒞}_A\right|-1,\forall {𝒞}_A\in 𝒞.$$ (23)

This formulation has exponentially many constraints.

Another way to rule out cycles is by imposing constraints such that the nodes follow a topological order (Park and Klabjan, 2017). A topological ordering is a unique ordering of the nodes of a graph from 1 to $m$ such that the graph contains an arc $(j,k)$ if node $j$ appears before node $k$ in the order. We refer to this formulation as topological ordering (TO). Define decision variables $o_{rs}\in \left\{\mathrm{0,\; 1}\right\}$ for all $r,s\in \{1,\dots ,m\}$. The variable $o_{rs}$ takes value 1 if the topological order of node $r$ equals $s$. The TO formulation rules out cycles in the graph by the following constraints

$$\mathbf{\text{TO }}1-m+m{g}_{kj}\le \sum _{s\in V}s\left(o_{ks}-o_{js}\right),\forall \left(j,k\right)\in \overrightarrow{E},$$ (24a)

$$\sum _{s\in V}{o}_{rs}=1\forall r\in V,$$ (24b)

$$\sum _{r\in V}{o}_{rs}=1\forall s\in V.$$ (24c)

This formulation has $𝒪\left(m^2\right)$ variables and $𝒪\left(\right|\overrightarrow{E}\left|\right)$ constraints.