Bayesian Regression with Network Prior: Optimal Bayesian Filtering Perspective

Abstract

The recently introduced intrinsically Bayesian robust filter (IBRF) provides fully optimal filtering relative to a prior distribution over an uncertainty class ofjoint random process models, whereas formerly the theory was limited to model-constrained Bayesian robust filters, for which optimization was limited to the filters that are optimal for models in the uncertainty class. This paper extends the IBRF theory to the situation where there are both a prior on the uncertainty class and sample data. The result is optimal Bayesian filtering (OBF), where optimality is relative to the posterior distribution derived from the prior and the data. The IBRF theories for effective characteristics and canonical expansions extend to the OBF setting. A salient focus of the present work is to demonstrate the advantages of Bayesian regression within the OBF setting over the classical Bayesian approach in the context otlinear Gaussian models.

I. Introduction

Owing to increased computational ability, there is a growing desire to model complex systems. Unfortunately, rarely is there sufficient data to obtain satisfactory system identification. The problem exists across many areas of science, engineering, economics, and social science. Even when a system is not overly complex, accurate system identification is often impossible due to a paucity of data. Hence, there is a growing need to model system uncertainty in the sense that there is uncertainty class of models, to which the underlying model is a member, and then to design robust operatorthat is optimal over the uncertainty class with respect to some cost function.

Consider biology, where there is interaction across tens of thousands of genes and proteins in a single cell and billions of cells interacting in a multi-scale fashion within a single organism [1]–[3]. One would like to model gene regulatory networks in order to design classifiers for disease diagnosis or intervention strategies for treatment [4]–[6]. Owing to model complexity, intervention may need to take uncertainty into account [7]–[9].

In signal processing, robust filtering dates back to the 1970s in the design of optimal linear filters with uncertain power spectra and minimax optimization [10]–[14]. Minimax robust filters have been studied for various kinds of filters, including matched [15] and Kalman filters [16]. Bayesian robust filtering was proposed first for binary filtering [17] and then for linear filtering [18]. In the Bayesian framework, a prior distribution is assumed over the uncertainty class and optimization is relative to that distribution. In [17] and [18], optimization is constrained to the filters that are optimal for some model in the uncertainty class. In [19] that restriction is lifted.

In this paper, we extend the robust filtering framework in [19] by considering posterior distributions resulting from incorporating new data with the prior distribution in the Bayesian framework. In particular, we are interested in the regression formulation from a Bayesian perspective. This will allow us to systematically quantify the uncertainty of the prior knowledge on the system under study when regression accuracy is the operational objective ofinterest. As we would like to have closed-form results in this paper, we make simplified model assumptions when needed.

The model formulation of optimal Bayesian regression in this paper is fundamentally different from the existing Bayesian regression literature, in which regression functions or models are pre-defined in certain functional forms, for example, Bayesian linear regression (BLR) [20]–[24]. Existing BLR models take the pre-defined functional formsY = ψ(X ; β) = β^T X from covariates X (denoting measurement data) to predict outcome response Y (denoting system characteristics). They typically enforce prior distribution assumptions on model coefficients β, including traditional multivariate Gaussian prior assumptions [20]–[23] as well as other sparse prior assumptions, including spike-and-slab prior, Bernoulli-Gaussian prior, Laplace prior, and other structured sparse prior assumptions arising from LASSO, group LASSO, and graphical LASSO [24]–[31]. In these existing BLR models, the connection of the regression functions and prior assumptions with the underlying systems or random processes is vague. There is a scientific gap in constructing functional models and making prior assumptions on model parameters when the actual uncertainty applies to the underlying random processes. Thus, given partial prior knowledge for the underlying random processes, the prior distribution should be placed directly on the random processes themselves, which is the approach taken in [32], [19], and here.

Our main contribution in this paper is to formulate Bayesian regression from the perspective of optimal Bayesian filtering (OBF), directly modeling the problem based on the joint distribution of covariatesX and outcome responseY . Instead of directly imposing model assumptions on model coefficients without systematic modeling ofinterdependence relationships of the involved random variables, as in classical Bayesian regression methods, our new optimal Bayesian regression, based on optimal Bayesian filtering theory, models the joint distribution of involved random variables, both covariates and responses. We emphasize that the prior on interdependence relationships in the OBF framework, rather than directly enforcing ad hoc prior assumptions on model coefficients of the pre-defined regression functions as in classical Bayesian regression, can help improve prediction accuracy when data are limited.

The remainder of the paper is organized as follows. In Section II, we review the robust filtering theory in [19]. Section III derives the general framework for optimal Bayesian filtering by extending the robust filtering theory. Section IV describes the proposed optimal Bayesian regression (OBR) via the OBF theory, with the linear Gaussian model being the network prior as an example. In Section V, we provide experimental results with simulated data as well as a network example motivated by modeling colon cancer to demonstrate that the OBR can naturally integrate the network prior and improve the regression accuracy when there are not enough data. Finally, we conclude the paper in Section VI.

II. Intrinsically Bayesian Robust Filters

Intrinsically Bayesian robust filtering extends classical filtering by optimizing over random processes in the presence of uncertainty regarding those processes. Optimal Bayesian filtering extends Bayesian robust filtering by including data and can directly apply results for Bayesian robust filtering to its own setting. This section briefly reviews some basics of Bayesian robust filtering as developed in [19].

The basic filtering problem involves jointly distributed observation and signal random processes, (X(t), Y(v)), t ∈ T, v ∈ V , with T and V being index sets, possessing a joint probability distribution function:

$$F(x,y;t,v)=P(X\left(t\right)\le x,Y\left(v\right)\le y)$$ (1)

If X is an p-dimensional random vector, then X ≤ x(t) is replaced by p component inequalities. To make notations simple, we write F(x, y; t, v) in either case. It is commonplace for the distribution of the process not to be known, in which case we have known or partially known statistical properties. Optimal signal estimation involves estimating a signal Y(v) at time v via a filter ψ given observations {X(t)}_t∈T, namely, $\widehat{Y}\left(\upsilon \right)=\psi \left(X\right)\left(\upsilon \right)$. Optimization is relative to a family of filters, ℱ, where ψ ∈ ℱ is a mapping ψ : 𝒪 → ℂ and 𝒪 is the space of possible observed signals, and a cost function $C(Y\left(\upsilon \right),\widehat{Y}\left(\upsilon \right))$. If an optimal filter (OF) exists for a fixed v ∈ V (with finite error), then it can be expressed as

$${\psi }_{\text{OF}}\left(X\right)\left(v\right)=\text{arg}\underset{\psi \in \mathcal{ℱ}}{\text{min}}C(Y\left(v\right),\psi \left(X\right)\left(v\right)),$$ (2)

where the minimum may be achieved by more than a single ψ ∈ ℱ. For the mean-square error (MSE) cost function,

$$C(Y\left(v\right),\psi \left(X\right)\left(v\right))={\text{E}}_F\left[{\mid Y\left(v\right)-\psi \left(X\right)\left(v\right)\mid }^2\right],$$ (3)

where E_F denotes the expectation with respect to F and an optimal filter is referred to as a minimum-mean-square-error (MMSE) filter.

When the model is not known with certainty, we assume that the joint process belongs to an uncertainty class of processes defined by the parameter set Θ, each θ ∈ Θ corresponding to a distribution F_θ(x, y; t, v), and optimality is defined relative to Θ. As characterized in [19], given an uncertainty class Θ, a prior distribution π(θ), a cost function C, and a filter class ℱ, an intrinsically Bayesian robust filter (IBRF) in ℱ relative to π is defined by

$${\psi }_{\text{IBRF}}\left(X\right)\left(v\right)=\text{arg}\underset{\psi \in \mathcal{ℱ}}{\text{min}}{\text{E}}_{\pi }\left[C(Y_{\theta }\left(v\right),\psi \left(X_{\theta }\right)\left(v\right))\right].$$ (4)

The adverb “intrinsically” refers to the fact that optimization is over all filters in ℱ, as opposed to only being taken with respect to filters that are optimal for some θ ∈ Θ, in which case an optimal filter is called a model-constrained Bayesian robust filter (MCBRF) [17], [18]. In the terminology of [33], the IBRF Bayesian framework is M-closed because the uncertainty class Θ consists of a parametric class of possible models.

A. Effective Characteristics

A key to finding an IBRF is that finding an optimal filter relative to a prior distribution over an uncertainty class of joint random processes often reduces to finding a single effective joint random process whose optimal filter is the IBRF for the uncertainty class [19].

Definition 1: Let C : ℂ × ℂ → ℝ⁺ be a measurable cost function of two variables, where ℝ⁺ is the set of non-negative real numbers. Let ℱ be a function class, where each ψ ∈ ℱ maps from a signal space, 𝒪, to ℂ. An observation and signal pair, (X(t), Y(v)), is solvable under cost C and function class ℱ if there exists a function ψ_OF ∈ ℱ minimizing C(Y(v), ψ(X)(v)) over all ψ ∈ ℱ.

Definition 2: Let Θ be an uncertainty class of process pairs having prior distribution π. An observation and signal pair (X_eff(t), Y_eff(v)) is an effective process under cost C, function class ℱ, and uncertainty class Θ if for all ψ ∈ ℱ, both E_π[C(Y_θ(v), ψ(X_θ)(v))] and C(Y_eff(v), ψ(X_eff)(v)) exist and

$${\text{E}}_{\pi }\left[C(Y_{\theta }\left(v\right),\psi \left(X_{\theta }\right)\left(v\right))\right]=C(Y_{\text{eff}}\left(v\right),\psi \left(X_{\text{eff}}\right)\left(v\right)).$$ (5)

Theorem 1: Let Θ be an uncertainty class of process pairs having the prior distribution π. If there exists a solvable effective process, (X_eff(t), Y_eff(v)), with optimal filter ψ_eff, then ψ_IBRF = ψ_eff.

The effective process may or may not belong to the uncertainty class. The general procedure is to establish a class, Φ, of solvable processes possessing known optimal filters and find a process {X_eff(t), Y_eff(s)} ∈ Φ, for which (5) is satisfied. According to Theorem 1, an IBR filter is given by the optimal filter corresponding to the effective process.

Given a class of filters, filter error and optimality often depend on only certain characteristics, a characteristic being a deterministic function derived from the random process. For instance, the MMSE linear filter depends on only second-order moments. In addition to effective processes, an IBRF can often be expressed in the same closed form as a model-specific optimal filter with the original characteristics replaced by effective characteristics [19].

Definition 3: A class, Λ, of process pairs, {X_λ(t), Y_λ(s)}, is reducible under cost C and function class ℱ if there exists a cost functional 𝒢 such that for each λ ∈ Λ and ψ ∈ ℱ,

$$C(Y_{\lambda }\left(s\right),\psi \left(X_{\lambda }\right)\left(s\right))=\mathcal{𝒢}({\omega }_{\lambda },{\kappa }_{\psi }),$$ (6)

where ω_λ is a collection of process characteristics and κ_ψ represents parameters for filter ψ.

Definition 4: A collection of characteristics, ω, is solvable in the weak sense under cost functional 𝒢 and function class ℱ if there exists a solution to

$${\psi }_{\text{weak}}=\text{arg}\underset{\psi \in \mathcal{ℱ}}{\text{min}}\mathcal{𝒢}(\omega ,{\kappa }_{\psi }).$$ (7)

Given a set of characteristics, ω, that are solvable in the weak sense, there is an optimal filter, ψ_weak, that possesses a functional, 𝒢(ω, κ_ψweak).

Definition 5: Let Θ be an uncertainty class of process pairs contained in a reducible class. The characteristic ω_eff is an effective characteristic in the weak sense under cost functional 𝒢, function class ℱ and uncertainty class Θ if for all ψ ∈ ℱ, both E_Θ[𝒢(ω_θ , κ_ψ)] and 𝒢(ω_eff , κ_ψ ) exist and

$${\text{E}}_{ϴ}\left[\mathcal{𝒢}({\omega }_{\theta },{\kappa }_{\psi })\right]=\mathcal{𝒢}({\omega }_{\text{eff}},{\kappa }_{\psi }).$$ (8)

Theorem 2: Let Θ be an uncertainty class of process pairs contained in a reducible class. If there exist weak-sense solvable weak-sense effective characteristics, ω_eff, with optimal filter ψ_eff, then ψ_IBRF = ψ_eff.

If there exists an effective process providing the effective characteristics, then these are said to be effective in the strong sense; otherwise, they are effective in the weak sense.

B. Canonical Expansions

Many problems in random processes reduce to finding an integral canonical expansion of a random process X(t). Examples include optimal filtering, signal detection, and Karhunen-Loéve compression. Assuming zero-mean processes, a canonical expansion takes the form

$$X\left(t\right)={\int }_{\Phi }Z\left(\xi \right)x(t,\xi )d\xi ,$$ (9)

where Z(ξ) is the white noise over Φ (the domain of ξ) and the coordinate functions x(t, ξ) are deterministic. Examples include optimal filtering, signal detection, and Karhunen-Loéve compression. The covariance function of continuous white noise is the generalized function I(ξ)δ(ξ − ξ′), where I(ξ) is the intensity of the white noise and the theory of integral representation is interpreted in the generalized sense. X(t) has the auto-correlation function

$$R_X(t,t^{\prime })={\int }_{\Phi }I\left(\xi \right)x(t,\xi )\overline{x(t^{\prime },\xi )}d\xi$$ (10)

and x(t, ξ) = R_XZ(t, ξ)I(ξ)⁻¹, where R_XZ(t, ξ) is the cross-correlation between X(t) and Z(ξ).

Integral canonical expansions are formed via a kernel a(t, ξ) by defining

$$Z\left(\xi \right)={\int }_{T}X\left(t\right)\overline{a(t,\xi )}dt.$$ (11)

Three conditions are necessary and sufficient for a canonical expansion to result [34], [35]. In [19], these conditions are extended to IBR filtering by taking the expectations of the involved covariances with respect to the prior distribution and take the form

$$x(t,\xi )=\frac{1}{I\left(\xi \right)}{\int }_{T}a(v,\xi ){\text{E}}_{\pi }\left[R_{X_{\theta }}(t,v)\right]dv,$$ (12)

$${\int }_T\overline{a(t,\xi )}x(t,{\xi }^{\prime })dt=\delta (\xi -{\xi }^{\prime }),$$ (13)

$${\int }_{\Phi }x(t,\xi )\overline{a(t^{\prime },\xi )}d\xi =\delta (t-t^{\prime }),$$ (14)

where the intensity of the white noise is given by

$$I\left(\xi \right)={\int }_{\Phi }{\int }_T{\int }_T{\text{E}}_{\pi }\left[R_{X_{\theta }}(t,t^{\prime })\right]\overline{a(t,\xi )}a(t^{\prime },{\xi }^{\prime })dtd{t}^{\prime }d{\xi }^{\prime }.$$ (15)

Consider linear filters of the form

$$\psi \left(X\right)\left(v\right)={\int }_{T}g(v,t)X\left(t\right)dt,$$ (16)

where the integral has a finite second moment. For effective process (X_eff(t), Y_eff(v)), if there exists an optimal MSE linear filter to estimate Y_eff via X_eff, then it is given by

$${\widehat{Y}}_{\text{eff}}\left(v\right)={\int }_T\widehat{g}(v,t)X_{\text{eff}}\left(t\right)dt,$$

where

$$\widehat{g}(v,t)={\int }_{\Xi }\overline{\frac{a(t,\xi )}{I\left(\xi \right)}}{\int }_T{\text{E}}_{\pi }\left[R_{Y_{\theta }{X}_{\theta }}(v,u)\right]a(u,\xi )dud\xi .$$ (18)

III. Optimal Bayesian Filters

As opposed to the setting in [19], suppose that, in addition to an uncertainty class and prior distribution, data are drawn from the joint random process to produce the random sample S = {(x(t₁), y(v₁)), … , (x(t_n), y(v_n))}, and π(θ, S) is the joint distribution over Θ and the sampling process. We refer to the operator providing the minimum expectation conditioned on the data as an optimal Bayesian filter (OBF):

$${\psi }_{\text{OBF}}(X,S)\left(v\right)=\text{arg}\underset{\psi \in \mathcal{ℱ}}{\text{min}}{\text{E}}_{\pi }[C(Y_{\theta }\left(v\right),\psi \left(X_{\theta }\right)\left(v\right))\mid S].$$ (19)

The conditional expectation is relative to the posterior density π^*(θ) = π(θ|S). When expressing the OBF in terms of the posterior, the expectation E_π [ · |S] in (26) is changed to the expectation E_π* [ · ]. The dependence of (X, Y) on θ is made explicit by writing (X_θ , Y_θ); should the parameterization not be explicit, then the expectation takes the form E_π[C(Y(v), ψ(X)(v))|θ|S] or E_π* [C(Y(v), ψ(X)(v))|θ]. The OBF terminology is analogous to the corresponding problem in classification, where an optimal classifier relative to the posterior over an uncertainty class of feature-label distributions is called an “optimal Bayesian classifier” [32].

Note that we are not postulating a filtering model from X(t) to Y(v) and placing a prior on the model parameters. Since the family {F_θ (x, y; t, v) : θ ∈ Θ} represents our incomplete knowledge regarding the processes, prior knowledge relates directly to the joint distributions in the uncertainty class. If there are no data, then the OBF reduces to the IBRF. Moreover, the OBF for π and S is the IBRF for the posterior π^*.

For the MSE cost function,

$${\psi }_{\text{OBF}}(X,S)\left(v\right)=\text{arg}\underset{\psi \in \mathcal{ℱ}}{\text{min}}{\text{E}}_{{\pi }^{\ast }}{\text{E}}_F[Y_{\theta }\left(v\right)-\psi \left(X_{\theta }\right)\left(v\right){\mid }^2].$$ (20)

If ℱ consists of all measurable functions, then the MSE optimal filter is given by the conditional expectation. Thus,

$$\begin{array}{cc}\hfill & {\psi }_{\text{OBF}}(X,S)\left(v\right)\hfill \\ \hfill & ={\text{E}}_{{\pi }^{\ast }}{\text{E}}_F[Y\left(v\right)\mid X,\theta ]\hfill \\ \hfill & ={\text{E}}_{{\pi }^{\ast }}{\text{E}}_F[Y\left(v\right)\mid X\mid \theta ]\hfill \\ \hfill & ={\text{E}}_{{\pi }^{\ast }}[{\psi }_{\text{OF}}\left(X\right)\left(v\right)\mid \theta ].\hfill \end{array}$$ (21)

Hence, the OBF can be found by taking the optimal filter (conditional expectation) for each θ and then averaging over the posterior.

A different view of the MSE OBF can be obtained by rewriting (21):

$$\begin{array}{cc}\hfill & {\psi }_{\text{OBF}}(X,S)\left(v\right)\hfill \\ \hfill & ={\text{E}}_{{\pi }^{\ast }}{\text{E}}_F\left[Y\left(v\right)\mid X\mid \theta \right]\hfill \\ \hfill & {\int }_{ϴ}{\int }_{-\infty }^{\infty }yd{F}_{\theta }(y\mid X){\pi }^{\ast }\left(\theta \right)d\theta \hfill \\ \hfill & {\int }_{-\infty }^{\infty }y\text{d}{\int }_{ϴ}{F}_{\theta }(y\mid X){\pi }^{\ast }\left(\theta \right)\text{d}\theta .\hfill \end{array}$$ (22)

Here the OBF is found by averaging the conditional distribution for Y given X over the posterior and then taking the conditional expectation of Y relative to that average. This average is called the effective conditional distribution for Y given X, relative to the posterior distribution π^*. It is denoted by $F_{\text{eff}}^S(y\mid X)$ and (22) becomes

$${\psi }_{\text{OBF}}(X,S)\left(v\right)={\int }_{-\infty }^{\infty }y\text{d}{F}_{\text{eff}}^S(y\mid X)={\text{E}}_{F_{\text{eff}}^S}[Y\mid X],$$ (23)

which is simply the usual conditional expectation form of the optimal MSE filter, but relative to the effective conditional distribution. The optimal MSE filter can be derived for any uncertainty class of models F.

To compute required statistical properties such as moments, we define the effective joint distribution of (X, Y) by

$$F_{\text{eff}}^S(x,y)={\int }_{ϴ}{F}_{\theta }(x,y){\pi }^{\ast }\left(\theta \right)\text{d}\theta .$$ (24)

In general, the expectation of the p-q moment relative to the posterior distribution can be computed by

$$\begin{array}{cc}\hfill & {\text{E}}_{{\pi }^{\ast }}{\text{E}}_F[X^p\left(t_a\right)Y^q\left(v_b\right)\mid \theta ]\hfill \\ \hfill & ={\int }_{ϴ}{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{x}^p\left(t_a\right)y^q\left(v_b\right)\text{d}{F}_{\theta }(x,y){\pi }^{\ast }\left(\theta \right)\text{d}\theta \hfill \\ \hfill & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{x}^p\left(t_a\right)y^q\left(v_b\right)\text{d}{F}_{\text{eff}}^S(x,y)\hfill \\ \hfill & ={\text{E}}_{F_{\text{eff}}^S}\left[X^p\left(t_a\right)Y^q\left(v_b\right)\right],\hfill \end{array}$$ (25)

which is the joint moment relative to the effective joint distribution.

Given the random sample S, the posterior distribution is given by

$$\begin{array}{cc}\hfill {\pi }^{\ast }\left(\theta \right)& =\pi (\theta \mid S)\hfill \\ \hfill & \propto \pi \left(\theta \right)\pi (S\mid \theta )\hfill \\ \hfill & =\pi \left(\theta \right)\prod _{i=1}^n{f}_{\theta }(x_i,y_i,;t_i;t_i,v_i),\hfill \end{array}$$ (26)

where we have assumed that F_θ(x_i, y_i; t_i, v_i) possesses the density f_θ(x_i, y_i; t_i, v_i). The constant of proportionality can be found by normalizing the integral of π^*(θ) to 1. It is important to recognize that in (26) the density f_θ(x_i, y_i; t_i, v_i) is not fixed in general and depends on the index i (unless the joint process happens to be stationary).

In general, π(θ) is not required to be valid density function. The priors are called “improper” if the integral of π(θ) is infinite, i.e., if π(θ) induces a σ-finite measure but not a finite probability measure. Such priors can be used to represent uniform weight for all parameters in an unbounded range, rather than truncating the range of each parameter to a finite range. The use of improper priors can lead to improper posterior distributions. When improper priors are used, Bayes’ rule does not apply, and (26) is taken as a definition, posterior distributions being normalized to possess unit integral. Hence, it is important to check that the normalizing integral does not diverge.

As an illustration, consider the simple one-dimensional case where T = V = {0} and F_θ(x, y; 0, 0) is Gaussian for all θ ∈ Θ as it is related to our latter derivation of optimal Bayesian regression with the network prior. Then

$${\text{E}}_F[Y\left(0\right)\mid X\left(0\right)=x\mid \theta ]={\mu }_{Y_{\theta \mid x}}={\mu }_{Y_{\theta }}+{\rho }_{\theta }\frac{{\sigma }_{Y_{\theta }}}{{\sigma }_{X_{\theta }}}(x-{\mu }_{X_{\theta }})$$

is the classical linear Gaussian regression, where μ_Xθ and μ_Yθ are the means and σ_Xθ and σ_Yθ are the standard deviations of X_θ and Y_θ, respectively. Hence, (21) yields

$$\begin{array}{cc}\hfill & {\psi }_{\text{OBF}}(X,S)\left(0\right)\hfill \\ \hfill & ={\text{E}}_{{\pi }^{\ast }}\left[{\mu }_{Y_{\theta }}\right]-{\text{E}}_{{\pi }^{\ast }}\left[{\rho }_{\theta }{\mu }_{X_{\theta }}\frac{{\sigma }_{Y_{\theta }}}{{\sigma }_{X_{\theta }}}\right]+{\text{E}}_{{\pi }^{\ast }}\left[{\rho }_{\theta }\frac{{\sigma }_{Y_{\theta }}}{{\sigma }_{X_{\theta }}}\right]x,\hfill \end{array}$$

which is still a linear function of x. In the OBF setting, the resulting regression function requires computing the effective statistical properties, which are the moments with respect to the posterior in this special case.

We emphasize that here the resulting linear regression function for predicting Y naturally comes from the Gaussian joint distribution assumption on F, which is fundamentally different from the classical Bayesian Linear Regression (BLR) that starts by assuming a linear regression function. In the classical BLR approach for this setup, a linear function of the form ψ(x) = β^Tx = β₀ + β₁x is assumed with a prior distribution on the parameter pair (β₀, β₁), thereby leading to a posterior distribution. The weakness of classical BLR is that the prior distribution is ad hoc relative to the underlying process F_θ(x, y; 0, 0), which in this case is a Gaussian. No matter the choice of prior for (β₀, β₁), the resulting regression can be no better than the OBF since, according to (21), the OBF is optimal. Structurally, the OBF is based on the uncertainty in the distribution of (X, Y ), which is characterized via π(θ), and the comparative performance of the BLR filter depends on the degree to which the prior on (β₀, β₁) happens to capture the knowledge in π(θ).

A. Canonical Expansions: Optimal Bayesian Wiener Filter

The IBRF theory for canonical expansions extends to IBRFs by replacing the prior π in (11) through (15) with the posterior π^*. We illustrate the use of canonical representations by deriving the optimal Bayesian Wiener filter via the theory of effective processes with respect to the posterior distribution. Let ℱ be the class of all linear integral operators of the form (16). The solvable class consists of all process pairs, (X, Y), such that ψ(X)(v) has a finite second moment for any g(v, t) and there exists $\widehat{g}(\upsilon ,t)$ for which the Wiener-Hopf equation is satisfied. Define R_Θ,Y(v, v) = E_π* [R_Y#x03B8;(v, v)], R_Θ,X(t, u) = E_π* [R_Xθ (t, u)], and R_{Θ,Y X}(v, t) = E_π* [R_YθXθ(v, t)]. Since R_Θ,X(t, u) satisfies the properties of a valid auto-correlation function, there exists a zero-mean Gaussian process X_eff with auto-correlation function R_Θ,X(t, u). Similarly there exists a zero-mean Gaussian process Y_eff with auto-correlation function R_Θ,Y (v, v) at v and cross-correlation function R_{Θ,Y X}(v, t) at v for all t ∈ T. Since (5) is satisfied, (X_eff(t), Y_eff(v)) is an effective process. Thus, if (X_eff, Y_eff) is a solvable process, then by Theorem 1 the OBF is found via $\widehat{g}(\upsilon ,t)$ in (18) with R_{Θ,Y X}(v, u) in place of E_π [R_YθXθ(v, u)].

Consider the usual Wiener setting in which the processes are wide-sense stationary (WSS) with auto-correlation functions r_Xθ (τ) and r_Y (τ), power spectral densities S_Xθ (ω) and S_Y (ω), and cross-correlation function r_{Yθ Xθ} (τ) having Fourier transform S_{Yθ Xθ} (ω). The effective process is solvable and has auto-correlation function r_Θ,X (τ) = E_π* [r_Xθ (τ)] and cross-correlation function r_{Θ,Y X} (τ) = E_π* [r_YθXθ (τ)]. The Fourier transforms of these are denoted by S_Θ,X (ω) and S_{Θ,Y X} (ω), respectively, and are called the effective power spectra. The classical theory applies to the effective process, so we know that it is solvable.

Letting T = (−∞, ∞) and a(t, ω) = e^jωt, (11) defines Z_eff(ω) in terms of X_eff(t). Equations (12) through (14) are satisfied for the effective process with π^* in place of π and X_eff(t) has the integral canonical expansion (9)

$$X_{\text{eff}}\left(t\right)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{Z}_{\text{eff}}\left(\omega \right)e^{i\omega t}d\omega .$$ (27)

From (15), Z_eff(ω) has intensity I_Θ(ω) = 2πS_Θ,X(ω). Substitution into (18) with τ = v − t yields

$${\widehat{g}}_{ϴ}\left(\tau \right)={\text{F}}^{-1}\left[\frac{S_{ϴ,YX}\left(\omega \right)}{S_{ϴ,X}\left(\omega \right)}\right]\left(\tau \right),$$ (28)

where F denotes the Fourier transform. According to Theorem 1, ${\widehat{g}}_{ϴ}\left(\tau \right)$ defines an OBF for Θ and is called the optimal Bayesian Wiener filter. It is of the same form as the classical Wiener filter but with the power spectra replaced by the effective power spectra relative to the posterior distribution, these being the effective characteristics for the OBF.

IV. Optimal Bayesian Regression for Networks

As discussed in Section III, when we consider regressing the outcome responses Y(v) on covariates X(t) given the available sample data S, the optimal Bayesian regression (OBR) with respect to MMSE is the conditional expectation E_π* E_F[Y(v)|X; θ]. We approach the problem from the perspective of a linear Gaussian model as our underlying system.

A. Linear Gaussian Models

Assume that the underlying regulations of a given network drive the network dynamics according to

$$X(t+1)=\mathbf{A}X\left(t\right)+ϵ,$$ (29)

in which t denotes discrete time points and ϵ ~ 𝒩 (0, Σ) is the noise. Note that the generality of A allows loops/cycles in the directed network model. At each time point t, the network dynamic evolves according to (29). This is a well-known time-homogeneous linear Gaussian model and X(t) is a first-order Gauss-Markov random process [36]. The main problem of this model is that, depending on the eigen-system structure of A, the network dynamic state X(t) may shrink or explode exponentially in the direction of the leading eigenvector of A. Hence, given a network topology, how can we can guarantee that X(t) has a steady-state distribution as t → ∞? When the leading eigenvalue is 1, the long-run behavior of the system is determined by the eigenvectors with the corresponding unit eigenvalue [36]. If we could have that guarantee, then X(t) follows a Gaussian distribution determined by A and Σ_t = cov(X(t)). To be specific,

$$cov\left(X(t+1)\right)=\mathbf{A}cov\left(X\left(t\right)\right){\mathbf{A}}^T+\sum =\mathbf{A}\sum _t{\mathbf{A}}^T+\sum .$$ (30)

If the system is stable, then the covariance also converges and there exists a positive semi-definite matrix Σ_∞ such that lim_t→∞ Σ_t = Σ_∞, which is the unique solution of the discrete-time Lyapunov equation

$$\sum _{\infty }=\mathbf{A}\sum _{\infty }{\mathbf{A}}^T+\sum$$ (31)

from the previous recursive equation [36]. Therefore, with prior knowledge on a given linear Gaussian model, the system’s steady-state behavior x = X(∞) follows a Gaussian distribution with the converging covariance matrix Σ_∞.

Finally, we assume that in real-world applications we are only interested in some outcome characteristics determined by the system’s steady-state behavior x:

$$y=Y\left(\infty \right)={\beta }^{T}X\left(\infty \right)+{ϵ}^{\prime }={\beta }^T\text{x}+{ϵ}^{\prime }.$$ (32)

Often β is a sparse vector. With this setup, we can derive the joint distribution F (x, y; ∞, ∞) and apply the OBF for optimal Bayesian regression (OBR) to estimate y given x.

We note here that we focus on regression problems with linear Gaussian models due to the closed-form OBR solutions resulting from the model assumption. However, the proposed OBR in the optimal Bayesian filtering framework can be generalized to non-Gaussian models for any family of joint distributions F(x, y; ∞, ∞) if they can be derived based on the prior systems knowledge, including the exponential family [37]. For example, as in Poisson regression, y is assumed to follow the Poisson distribution: y ~ Poisson(λ) with the rate parameter λ dependent on x through the canonical link function λ = exp(β^T x). However, even such a modest extension from linear Gaussian models will make the computation of the OBF difficult as it is well-known that the posterior can not be obtained in closed form for Bayesian Poisson Regression [38], [39]. We have to resort Monte Carlo Markov Chain (MCMC) methods to estimate effective characteristics for finding the OBF. It is more critical when we study multivariate exponential family joint distributions F(x, y; ∞, ∞), for which some caution may be exercised as they may not have closed and/or tractable forms, for example multivariate Poisson distributions [40]. In this paper, we focus on linear Gaussian models to illustrate the idea of OBF-based regression.

B. How does a network prior help regression?

We examine whether, and how, a priori network prior, such as the linear Gaussian model assumption, on X(t) provides better regression on predicting y considering the steady-state behavior when t → ∞?

1) Bayesian Linear Regression (BLR)

First consider the commonly adopted Bayesian linear regression (BLR) framework [21]–[30]. In BLR, given the sample S = {(x¹, y¹), … , (xⁿ, yⁿ)}, β is typically assumed to follow a prior distribution. For taking a conjugate prior, β is assumed to follow a Gaussian distribution, for example, p(β) ~ 𝒩 (0, σ²I). BLR computes the posterior

$$\begin{array}{cc}\hfill p(\beta \mid S)& \propto p\left(\beta \right)p(S\mid \beta )\hfill \\ \hfill & \propto p\left(\beta \right)\prod _{i}p(y^i,{\text{x}}^i\mid \beta )\hfill \\ \hfill & \propto p\left(\beta \right)\prod _{i}p(y^i\mid {\text{x}}^i;\beta )p({\text{x}}^i\mid \beta ).\hfill \end{array}$$ (33)

As xⁱ and β are independent, it is clear that the prior distributions p(β) and p(xⁱ) factorize. Hence, the prior knowledge on x, such as the linear Gaussian model assumption, does not affect the posterior distribution p(β|S); hence, it does not lead to better prediction.

2) Optimal Bayesian Regression (OBR) through OBF

Now consider OBR. Instead of blindly imposing the linear model and the prior distribution assumptions on the regression model coefficients β as in BLR, we directly investigate the joint distribution F(x, y; t, v) and derive the estimates based on it. As we are interested in steady-state behavior by the joint distribution p(x, y) = F(x, y; ∞, ∞), based on the linear Gaussian model, we know p(x) ~ 𝒩 (μ(X(0), A), Σ(A)) with both the mean and covariance depending on the network model A. On the other hand p(y|x) ~ 𝒩 (β^T x, σ^′2). Hence the joint distribution p(x, y) ~ 𝒩 (μ, Σ), in which θ = {μ, Σ} parametrizes the uncertainty class constrained by β, A, and X(0). To be more specific, based on the linear Gaussian model we can derive [36]

$$\begin{array}{cc}\hfill \mu & ={[{\mu }_{\text{x}},{\mu }_y]}^T={[\stackrel{‒}{\mathrm{x}},{\beta }^T\stackrel{‒}{\mathrm{x}}]}^T;\hfill \\ \hfill \sum & =\left[\begin{array}{cc}\hfill {\Sigma }_{\text{xx}}\hfill & \hfill {\Sigma }_{\text{xy}}\hfill \\ \hfill {\Sigma }_{y\text{x}}\hfill & \hfill {\Sigma }_{yy}\hfill \end{array}\right]=\left[\begin{array}{cc}\hfill {\Sigma }_{\infty }\hfill & \hfill {\Sigma }_{\infty }\beta \hfill \\ \hfill {\beta }^T{\Sigma }_{\infty }\hfill & \hfill {\beta }^T{\Sigma }_{\infty }\beta +{\sigma }^{\prime 2}\hfill \end{array}\right],\hfill \end{array}$$

in which $\stackrel{‒}{x}$ is determined by the eigen-system of A and X(0) of the linear Gaussian model, and Σ_∞ is the converging covariance matrix for the stable linear Gaussian model, satisfying the linear equation (31). Note that in practice we do not have these relationships between Σ and the model parameters. The prior knowledge for the joint distribution p(x, y) = F(x, y; ∞, ∞) will be directly on θ = {μ, Σ} rather than individual model parameters, including β, A, and X(0). The prior model with the uncertainty class of joint distributions governed by μ and Σ may come from accumulated scientific understanding of the system of interest, for example, the underlying systems model. In the present work, to get closed-form solutions, we focus on linear Gaussian models and demonstrate that our OBR through the OBF theory can naturally incorporate such network prior knowledge and improve the prediction accuracy compared to the classical Bayesian regression methods, especially when there are not enough data.

Similarly as the derived simple one-dimensional case in Section III, we have the following conditional expectation as the MSE optimal filter for multivariate Gaussian processes with p(x, y) being jointly Gaussian:

$${\text{E}}_F[y\mid \text{x};\theta ]={\mu }_y+\sum _{y\text{x}}\sum _{\text{xx}}^{-1}(\text{x}-{\mu }_{\text{x}}),$$ (34)

which is a linear function of x. We emphasize here again that such a linear OBR naturally emerges from the linear Gaussian model assumption. As derived in Section III, the MMSE OBR with the uncertainty class by θ = {μ, Σ} is the expectation

$${\psi }_{OBR}\left(\text{x}\right)={\psi }_{OBF}\left(\text{x}\right)=E_{{\pi }^{\ast }}{E}_F[y\mid \text{x};\theta ].$$ (35)

Hence,

$${\psi }_{OBR}\left(\text{x}\right)={\text{E}}_{{\pi }^{\ast }}\left[{\mu }_y\right]-{\text{E}}_{{\pi }^{\ast }}\left[\sum _{y\text{x}}\sum _{\text{xx}}^{-1}{\mu }_{\text{x}}\right]+{\text{E}}_{{\pi }^{\ast }}\left[\sum _{y\text{x}}\sum _{\text{xx}}^{-1}\right]\text{X}.$$ (36)

This resulting OBR ψ_OBR(x) under the linear Gaussian model assumption is indeed a linear regression function with respect to x with the regression model parameter estimates ${\widehat{\beta }}^T={\text{E}}_{\pi \ast }\left[{\sum }_{y\mathrm{x}}{\sum }_{\mathrm{xx}}^{-1}\right]$. The task here is to derive the moments with respect to the posterior distribution of uncertain models as effective characteristics. Clearly, the prior knowledge regarding the covariance of x may help derive better estimates ${\text{E}}_{\pi \ast }\left[{\sum }_{y\mathrm{x}}{\sum }_{\mathrm{xx}}^{-1}\right]$ for a better regressor. As an extreme case without observations, the correct network model may lead to better estimates of involved parameters and hence give better prediction.

The main message of this paper is to illustrate how prior network knowledge may help derive better regression models when there is a limited number of samples. To illustrate this main idea, we specifically focus on the uncertainty parameter Σ_xx without making assumptions on either β or Σ_yx directly. Hence, with the previous linear Gaussian model assumption, OBR resorts to the Bayesian estimates of the precision matrix $\left[{\sum }_{\mathrm{xx}}^{-1}\right]$ in classical Bayesian learning [20], [21], [24]. With x being multivariate Gaussian, a common conjugate prior for the model parameters for a multivariate Gaussian is to assume that the mean μ_x and the precision matrix $\Lambda ={\sum }_{\mathrm{xx}}^{-1}$ are Gaussian-Wishart [21]. Without loss of generality, we focus on the Bayesian estimate of the precision matrix ${\text{E}}_{\pi \ast }\left[\Lambda \right]={\text{E}}_{\pi \ast }\left[{\sum }_{\mathrm{xx}}^{-1}\right]$. When we have the network prior on x as the linear Gaussian model $\mathrm{x}~\mathcal{𝒩}(\stackrel{‒}{x},{\sum }_{\infty })$ we can take the following Wishart distribution as the prior for the precision matrix Λ [21], [24]:

$$\Lambda ~\text{Wi}(\Xi ,\nu )=\frac{{\mid \Lambda \mid }^{\frac{\nu -p-1}{2}}}{2^{\frac{\nu p}{2}}{\Gamma }_p\left(\frac{\nu }{2}\right){\mid \Xi \mid }^{\frac{\nu }{2}}}\text{exp}\left(-\frac{1}{2}\text{tr}(\Lambda {\Xi }^{-1})\right)$$ (37)

in which the conjugate prior Wishart distribution has two hyper-parameters, ν (degree of freedom) and Ξ (scale matrix), and p is the dimension of x with ν > p − 1. With the linear Gaussian network prior with the converging covariance matrix Σ_∞, Ξ can be set to (ν − p − 1)Σ_∞. Given the sample data S with n sample points with zero mean, the posterior distribution of the precision matrix can be derived as another Wishart distribution with the new parameters $\upsilon +\frac{n}{2}$ and $\Xi +\frac{n}{2}{\Xi }_S$:

$${\Lambda }_{{\pi }^{\ast }}~\text{Wi}(\Xi +\frac{n}{2}{\Xi }_S,\nu +\frac{n}{2}),$$ (38)

where ${\Xi }_S={\sum }_i{\mathbf{x}}^i{\mathbf{x}}^{iT}=D^{T}D$ is the estimated scatter matrix and D is the design matrix with the stacked xⁱ’s from the training samples S. Via this posterior distribution, we can now estimate the desired effective characteristics [21]:

$${\text{E}}_{{\pi }^{\ast }}\left[\sum _{\text{xx}}^{-1}\right]={\text{E}}_{{\pi }^{\ast }}[\Lambda ]=(\nu -p-1+\frac{n}{2})inv(\Xi +\frac{n}{2}{\Xi }_S).$$ (40)

This leads to the corresponding OBR assuming non-informative priors on the other model parameters in θ = {μ, Σ}:

$${\psi }_{OBR}\left(\text{x}\right)\propto {\left[inv(\Xi +\frac{n}{2}{\Xi }_S)D^T\text{y}\right]}^T\text{x},$$ (40)

with absorbed corresponding constants based on the selection of the degree-of-freedom parameter ν, and here y denotes the stacked response vector of yⁱ’s from the training samples S. Clearly, in the OBR, the linear Gaussian network prior on x contributes to the final regression model and is expected to help enhance the prediction accuracy, especially when there are a limited number of training samples.

With the multivariate Gaussian assumption on the joint distribution p(x, y) = F(x, y; ∞, ∞), the OBR ends up with the linear regression model. To better connect the OBR under the Gaussian assumption and classical BLR methods, we study the desired regression model by directly investigating the characteristics of the outcome response, y = β^T x + ϵ′, to estimate E_π* E_F [yx] and E_π* E_F [(β^T x+ϵ′)x] with respect to both sides of the model equation. Denoting cov^*(y, x) = E_π* E_F [yx] as our effective characteristics, we have

$$co{v}^{\ast }(y,\text{x})=co{v}^{\ast }({\beta }^T\text{x}+{ϵ}^{\prime },\text{x}).$$ (41)

With a relatively weak assumption that E_F [ϵ′|x] = 0, it is obvious that cov^*(ϵ′, x) = 0. Therefore,

$$\begin{array}{cc}\hfill co{v}^{\ast }(y,\text{x})& =co{v}^{\ast }({\beta }^T\text{x}+{ϵ}^{\prime },\text{x})\hfill \\ \hfill & =co{v}^{\ast }\left({\beta }^T\text{x,x}\right)+co{v}^{\ast }({ϵ}^{\prime },\text{x})\hfill \\ \hfill & =co{v}^{\ast }\left(\text{x,x}\right)\beta .\hfill \end{array}$$ (42)

Hence, by straightforward algebraic manipulations,

$$\beta =inv\left(co{v}^{\ast }\left(\text{x,x}\right)\right)co{v}^{\ast }(y,\text{x}).$$ (43)

Along this direction, so far we have not imposed any distributional assumptions on x. What interests us is whether any prior knowledge on x helps better prediction as in our OBR (40). In (43), we see clearly that the parameter estimates of β depend on whether the covariance estimates cov^*(y, x) and cov^*(x, x) are accurate, which again turns towards the estimates of the second-order moments with respect to the posterior as effective characteristics. Hence, better estimates of cov^*(x, x) may help better model inference and therefore lead to better prediction accuracy. Especially, with small sample size compared to an extremely high number of covariates x, it is difficult to achieve accurate and robust estimates of the covariance matrix of x due to the quadratic number of the involved parameters with respect to the dimension of x. Hence, we expect that prior knowledge on the covariance matrix can help provide more stable and accurate estimates of β.

From the previous reasoning, the critical step for the OBR is to estimate

$${\psi }_{OBR}\left(x\right)={\widehat{\beta }}^T\text{x}=E_{{\pi }^{\ast }}\left[\sum _{y\text{x}}\sum _{\text{xx}}^{-1}\right]\text{x}.$$ (44)

As discussed, when we only impose prior knowledge with respect to Σ_xx and assume the zero-mean joint Gaussian distribution with μ_x and μ_y being zero, the MMSE OBR with the uncertainty class by θ = {Σ_xx} is the expectation ${\psi }_{OBR}\left(\mathbf{x}\right)={\widehat{\beta }}^T\mathbf{x}$ given by (40). Based on (42), deriving $\widehat{\beta }$ requires accurate estimation of the inverse of the expected covariance matrix

$$inv\left(co{v}^{\ast }\left(\text{x,x}\right)\right)=E_{\pi }^{-1}\left[\sum _{\text{xx}}\right]=\sum _{{\pi }^{\ast }}^{-1}.$$ (45)

Due to the inherent relationships between Wishart distributions and Inverse-Wishart (IW) distributions [21], [24], it can be shown that we will have an equivalent regression model of (40) through (42). Indeed, the derivation of $\widehat{\beta }$ resorts to the Bayesian Covariance Estimates (BCE) in classical Bayesian learning [20], [21], [24] with the previous linear Gaussian model assumption. For BCE, we assume that the conjugate prior for the model parameters μ_x and Σ_xx is Gaussian-Inverse-Wishart (GIW). Focusing on the covariance matrix Σ_xx, we can take the following Inverse-Wishart distribution as the prior:

$$\Sigma ~\text{IW}(\Xi ,\nu )=\frac{{\mid \Xi \mid }^{\frac{\nu }{2}}}{2^{\frac{\nu p}{2}}{\Gamma }_p\left(\frac{\nu }{2}\right)}{\mid \Sigma \mid }^{-\frac{\nu +p+1}{2}}\text{exp}\left(-\frac{1}{2}\text{tr}(\Xi \sum ^{-1})\right),$$ (46)

with two hyper-parameters ν and Ξ. Similarly, Ξ can be set to incorporate the available network prior on converging co-variance Σ_∞. The posterior distribution of Σ_xx given training sample S can be derived as $\mathbf{IW}(\Xi +\frac{n}{2}{\Xi }_{S,\upsilon }+\frac{n}{2})$.

We can now first estimate the desired effective characteristics, cov^*(x, x) = Σ^π*. For example, either the Maximum A Posterior (MAP or mode) estimate

$$\sum _{map}=\frac{\Xi +\frac{n}{2}{\Xi }_S}{\nu +p+1+\frac{n}{2}}$$ (47)

or the mean estimate

$$\sum _{mean}=\frac{\Xi +\frac{n}{2}{\Xi }_S}{\nu -p-1+\frac{n}{2}}$$ (48)

can be used as the appropriate estimates for Σ^π*. We then can derive the corresponding parameter estimate for the OBR under linear Gaussian models:

$$\widehat{\beta }\propto inv(\frac{n}{2}{\Xi }_S+\Xi )D^T\text{y}=inv(\frac{n}{2}{D}^{T}D+\Xi )D^T\text{y},$$ (49)

with absorbed corresponding constants depending on the choice of ν and whether we adopt the mode estimate Σ_map in (47) or mean estimate Σ_mean in (48). It is clear that the OBR (40) and (49) are equivalent with appropriate hyper-parameter setups, specifically, by setting the degree-of-freedom parameter ν to make sure the contributions from the prior knowledge Σ_∞ and training samples Ξ_S have the same weights. Note that taking the prior scatter matrix in the Inverse-Wishart conjugate prior as a diagonal matrix λI yields the equivalent solution to the BLR with Gaussian prior of the specific diagonal covariance assumption, whose Maximum Likelihood Estimate (MLE) results in the ridge regression formulation.

In summary, with appropriate prior knowledge on the linear Gaussian model for x, we expect better model estimates for the regression problem, especially with small sample sizes.

V. Experiments and Discussion

A. Simulations with Linear Gaussian Networks

Simulation experiments based on stable linear Gaussian models have been implemented to demonstrate the improvement of OBR with network prior knowledge over classical BLR. We randomly generate linear Gaussian networks by randomly simulating A, with rejection sampling to keep only stable models with the magnitude of the leading eigen-value less than or equal to 1. With that we are guaranteed stable X(∞). We simulate the network dynamics with A and take the steady-state samples X(∞) upon reaching the converging covariance matrix. Then we simulate the outcome y = β^T X(∞) + ϵ′, where β ~ 𝒩 (0, σ²I) and ϵ′ ~ 𝒩 (0, σ²) with σ = 1.

We have simulated data based on both the linear Gaussian network model with five nodes and linear regression for y. Based on the implementations of the BLR methods in the Probabilistic Modeling ToolKit (PMTK3: https://github.com/probml/pmtk3) [24], we have compared the MSE from both the OBR and classical BLR methods with different prior assumptions based on 1,000 simulated test samples using different training sample sizes. Our aim is to demonstrate that the prior knowledge of the linear Gaussian model (on x only) can help better predict y in OBR. Note that we assume no prior model knowledge on the regression function parameters β in both our OBR implementation and BLR. For BLR, we have tested non-informative Jeffreys prior [41], [42], Zellner’s g-prior [43], [44], as well as Gaussian prior by assuming diagonal Gaussian covariance matrix for the regression coefficient β, which corresponds to ridge regression for its MLE solution [21], [22], [24], [45]. For the hyper-parameters in these models, we have taken empirical Bayesian estimates. We note that for Jeffrey’s prior, performance is quite unstable with higher MSE when the sample size is fewer than the number of covariates. For such cases with much worse performance than OBR, including Zellner’s g-prior, we do not plot the results. As illustrated in Figure 1, OBR performs consistently better than BLR methods, especially with small samples (≤ 10). When the sample size is close to 20, different methods achieve similar MSE performance. All methods converge to the Bayes error (1.0) when the number of samples increases.

Figure 1.

Performance comparison between OBR and BLR for linear regression with covariates from a linear Gaussian network model with five nodes.

Based on the simulation results, it is clear that the interdependence relationships among measurements x themselves as well as the interrelationships between x and y affect the solutions. In BLR, as the derivation is based on the factorization p(y|x; β) and p(β), where β denotes the linear model parameters, the prior knowledge on the interdependence relationships among x may not help improve the predictive models. OBR is fundamentally different from BLR and is naturally derived based on the multivariate Gaussian prior knowledge considering all the interdependence relationships, which leads to more accurate and robust prediction.

B. A Colon Cancer Example

We now consider on phenotypic prediction with pathway models and compare OBR performance with BLR methods. Motivated by modeling colon cancer disease mechanisms as described in [46], [47], we assume that the simplified regulatory model illustrated in Figure 2, abstracted from three basic pathways, Ras/Raf/Mek pathway, PI3K pathway, and JAK/STAT pathway, can approximately model dynamic genome behavior of colon cancer. In Figure 2, we model the regulatory relationships among 11 proteins or protein complexes: EGF, Ras, MEK1/2, PIK3CA, STAT3, mTORC1, HGF, IL6, PKC, SPYR4, and TSC1/TSC2. System dynamics are represented by the vector z = [EGF, HGF, IL6, Ras, PIK3CA, STAT3, TSC1/TSC2, mTORC1, SPYR4, PKC, MEK1/2]. As MEK1/2 is a typical downstream marker for colon cancer, we consider the expression level of MEK1/2 is indicative of colon cancer status. Hence, in the regression setup, x = [EGF, HGF, IL6, Ras, PIK3CA, STAT3, TSC1/TSC2, mTORC1, SPYR4, PKC] and the disease outcome y = [MEK1/2]. Here, EGF, HGF, and IL6 are upstream ligands or stimulation factors regulating downstream dynamics. We assume that the network dynamics follow a linear Gaussian network model, which has been one of the commonly adopted gene regulatory network models due to its simplicity [48]–[58]. To assign the corresponding model parameters, we first set the regulatory relationships among these entities as derived from the prior pathway knowledge described in [46]:

$$\begin{array}{cc}\hfill \text{Ras}& =\frac{1}{3}\text{EGF}+\frac{1}{3}\text{HGF}+\frac{1}{3}\text{IL}6+ϵ;\hfill \\ \hfill \text{PIK}3\text{CA}& \frac{1}{2}\text{HGF}+\frac{1}{2}\text{Ras}+ϵ;\hfill \\ \hfill \text{STAT}3& \frac{1}{3}\text{EGF}+\frac{1}{3}\text{IL}6-\frac{1}{3}\text{PIK}3\text{CA}+ϵ;\hfill \\ \hfill \text{TSC}1\text{TSC}2& ={ϵ}^{\prime };\hfill \\ \hfill \text{mTORC}1& =-\text{TSC}1∕\text{TSC}2+ϵ;\hfill \\ \hfill \text{SPYR}4& =\frac{1}{2}\text{STAT}3+\frac{1}{2}\text{mTORC}1+ϵ;\hfill \\ \hfill \text{PKC}& =\frac{1}{2}\text{IL}6-\frac{1}{2}\text{SPYR}4+ϵ;\hfill \\ \hfill \text{MEK}1∕2& =\frac{1}{2}\text{Ras}+\frac{1}{2}\text{PKC}+ϵ,\hfill \end{array}$$

TSC1/TSC2 is stuck at 0 with a small probability of being changed: ϵ′ ~ 𝒩 (0, 0.05). This setup is motivated by the finding that the mutation for the tumor suppressor complex TSC1/TSC2 increases the colon cancer risk [46], [59]. For the upstream entities, [EGF, HGF, IL6] ~ 𝒩 (μ, Σ) with μ = [−0.3, −0.3, −0.3] and

$$\Sigma =\left[\begin{array}{ccc}\hfill 2.0\hfill & \hfill 0.4\hfill & \hfill 0.4\hfill \\ \hfill 0.4\hfill & \hfill 2.0\hfill & \hfill 0.4\hfill \\ \hfill 0.4\hfill & \hfill 0.4\hfill & \hfill 2.0\hfill \end{array}\right]$$

Finally, ϵ ~ 𝒩 (0, 0.2). This simplified colon cancer model is a linear Gaussian network approximation. From the assumed dependencies, we can construct the covariance matrix of all the involved entities. The task is to show that OBR outperforms traditional Bayesian regression methods when estimating the expression level of MEK1/2.

Figure 2.

A simplified colon cancer network [46]: regular arrows represent activating regulations while blunt arrows represent repressing regulations. To approximately model colon cancer, TSC1/TSC2 is stuck at “0”.

Analogous to the previous set of experiments, we simulate data based the colon cancer network model and compare the MSE for OBR and classical BLR methods with different prior assumptions based on 1,000 simulated test samples using different sample sizes. When testing OBR, we have perturbed the theoretically derived system covariance matrix to be the scale matrix of the Wishart prior distribution, since in reality we often do not have a complete and accurate network prior. For BLR, we have tested non-informative Jeffreys prior and Gaussian prior with empirical Bayesian estimates for the required hyper-parameters. As illustrated in Figure 3, OBR performs significantly better than BLR methods with limited sample sizes (≤ 40), which are typical in genomic applications [60]. All of the methods converge to the Bayes error (0.2) when the sample size increases. With small sample sizes, OBR with the network prior can provide more accurate and stable estimates.

Figure 3.

Performance comparison between OBR and BLR for the simplified colon cancer network model: (A) with all tested training sample size; (B) with the limited number of samples from 5 to 45.

VI. Conclusions

Given sample data, the theory of intrinsically Bayesian robust filters extends to optimal Bayesian filtering simply by forming a posterior distribution from the prior distribution and the data. The IBRF theory for effective characteristics and canonical expansions directly extends to the OBF setting. The main issue addressed here has been to demonstrate the advantage of regression (OBR) within that setting over the classical Bayesian regression in the context of linear Gaussian models.

We plan to generalize the essential idea to develop new network models and OBR for non-Gaussian random processes. When the family of joint distributions is complicated, we cannot expect to derive closed-from solutions for model parameter updates for the posterior distribution and OBR. As in the case of optimal Bayesian classification [32], MCMC methods will be utilized [38], [39].

Finally, we note that our OBR has the critical assumption that there exists complete or partial knowledge on the underlying systems to achieve better prediction performance. As discussed in [33], we adopt an M-closed Bayesian framework. However, the prior knowledge is often an approximation to the true underlying systems. With the OBF framework that we develop here, we may quantify the uncertainty of the imposed model with the operative objective of regression accuracy to derive the corresponding guidelines for the potential iterative network prior construction and experimental design tasks based on the criterion of the Mean Objective Cost of Uncertainty (MOCU) [9]. Such extensions of OBR with M-open perspectives, for example by considering the uncertainty class of all models constrained by a Kullback-Leibler ball given a nominal joint distribution F(x, y; t, v) [33], are admittedly challenging and will be studied in our future research.

Biographies

Xiaoning Qian (S’01–M’07) received the Ph.D. degree in Electrical Engineering from Yale University, New Haven, CT, in 2005. Currently, he is an assistant professor with the Department of Electrical & Computer Engineering, Texas A&M University, College Station, TX. He is affiliated with the Center for Bioinformatics & Genomic Systems Engineering (CBGSE) and the Center for Translational Environmental Health Research (CTEHR) at Texas A&M. He also is a courtesy assistant professor in the Department of Computer Science & Engineering and the Department of Pediatrics at the University of South Florida, Tampa FL. He was with the Bioinformatics Training Program at Texas A&M University, sponsored by the National Cancer Institute (NCI). His recent honors include the US NSF CAREER Award, Montague-Center for Teaching Excellence Scholar at Texas A&M, and the Best Paper Award at the 11th Asia Pacific Bioinformatics Conference. His current research interests include computational network biology, genomic signal processing, and biomedical image analysis.

Edward R. Dougherty is a Distinguished Professor in the Department of Electrical and Computer Engineering at Texas A&M University in College Station, TX, where he holds the Robert M. Kennedy ‘26 Chair in Electrical Engineering and is Scientific Director of the Center for Bioinformatics and Genomic Systems Engineering. He holds a Ph.D. in mathematics from Rutgers University and an M.S. in Computer Science from Stevens Institute of Technology, and has been awarded the Doctor Honoris Causa by the Tampere University of Technology. He is a fellow of both IEEE and SPIE, has received the SPIE President’s Award, and served as the editor of the SPIE/IS&T Journal of Electronic Imaging. At Texas A&M University he has received the Association of Former Students Distinguished Achievement Award in Research, been named Fellow of the Texas Engineering Experiment Station and Halliburton Professor of the Dwight Look College of Engineering. Prof. Dougherty is author of 16 books and author of more than 300 journal papers.