Estimation of the Distribution of Random Parameters in Discrete Time Abstract Parabolic Systems with Unbounded Input and Output: Approximation and Convergence

Abstract

A finite dimensional abstract approximation and convergence theory is developed for estimation of the distribution of random parameters in infinite dimensional discrete time linear systems with dynamics described by regularly dissipative operators and involving, in general, unbounded input and output operators. By taking expectations, the system is re-cast as an equivalent abstract parabolic system in a Gelfand triple of Bochner spaces wherein the random parameters become new space-like variables. Estimating their distribution is now analogous to estimating a spatially varying coefficient in a standard deterministic parabolic system. The estimation problems are approximated by a sequence of finite dimensional problems. Convergence is established using a state space-varying version of the Trotter-Kato semigroup approximation theorem. Numerical results for a number of examples involving the estimation of exponential families of densities for random parameters in a diffusion equation with boundary input and output are presented and discussed.

1. Introduction

The work we report on here was motivated by a compound inverse or blind deconvolution problem involving the interpretation of data from a transdermal alcohol biosensor. The observation (dating back to the 1930s [25, 35, 36, 37, 38]) that ethanol is highly miscible and finds its way into all the water in the body, and in particular, sweat, has in the past two decades, led to the development of technology to measure the amount of ethanol excreted from the body transdermally (i.e. through the skin) through perspiration and to then use it to quantitatively assess intoxication level. The basis for the measurement is an oxidation-reduction (redox) reaction that produces four electrons for each ethanol molecule oxidized. This results in a continuous current whose level is proportional to the amount of ethanol evaporating from the surface of the skin beneath the sensor. Now while these devices have been available and in use, both experimentally and commercially, for a number of years, they have been used primarily as abstinence monitors because transdermal alcohol level or concentration (TAC) data cannot consistently be converted to breath and blood alcohol concentrations (BrAC/BAC) across individuals, devices, and environmental conditions. (BAC and BrAC are currently, and historically have been, the standard measures of intoxication among alcohol researchers and clinicians, as well as in the courts.) Indeed, unlike a breath analyzer, which relies on a relatively simple model from basic chemistry (i.e., Henrys Law) for the exchange of gases between circulating pulmonary blood and alveolar air (see, for example, [22]) that has been found to be reasonably robust across the population, the transport and filtering of alcohol by the skin is physiologically more complex and is affected by a number of factors that differ across individuals (e.g., skin layer thickness, porosity and tortuosity, etc.) and even drinking episodes within individuals (e.g., body and ambient temperature, skin hydration, vasodilation). The challenge in making these devices practicable is to develop a means to reliably convert biosensor measured TAC into BAC or BrAC.

In our earlier work ([14, 19, 28]) we have taken a strictly deterministic approach to converting TAC to either BAC or BrAC. We fit first principles physics-based models in the form of a distributed parameter (diffusion) system with unbounded input and output, and used individual calibration data to capture the dynamics of the forward process - the propagation of alcohol from the blood, through the skin, and its measurement by the sensor (i.e. the forward model) by estimating the parameters (diffusivity, input/output gain, propagation inertia, etc.) that appear in the model via nonlinear least squares. Then in a second phase of processing, we use the fit model to deconvolve BAC or BrAC from the TAC signal measured by the biosensor in the field. However, because of the challenges described above, this approach was not entirely satisfying. Indeed, while it was possible to fit the models quite well to any particular drinking episode, we observed significant variance in the values of the parameters across different individuals and across different drinking episodes for the same individual. Consequently, the fit models did not yield the desired level of accuracy when they were used to deconvolve BAC or BrAC from TAC for a drinking episode that they were not specifically trained on.

To deal with this problem we have been looking at the idea of fitting a population forward model (having BAC or BrAC as input and TAC as output) in the form of a random partial differential equation, to data from multiple drinking episodes and multiple individuals and then using the population model to solve the deconvolution problem. Fitting a population model of this form implies that rather than estimate particular values for the parameters, we treat the parameters as random variables and estimate their distributions. In this way, it will become possible to produce not only an estimate for the BAC or BrAC, but also some form of credible bands to go along with it providing a quantitative estimate of the level of uncertainty in the estimate.

The basic underlying assumption in such an approach is that our first principles physics/physiological based model in essence, describes the dynamics common to the entire population (population interpreted broadly here to include not only all individuals, but also all devices, environmental conditions, and in effect, all ethanol molecules) and to then attribute all unmodeled sources of uncertainty (primarily due to variations in physiology, hardware, and the environment) observed in individual data to random effects. Moreover, we assume that what we observe in any individual data set is the combination or average of these random effects. Thus, this approach is realized by letting the parameters in the PDE model be random variables, the distributions of which are to be estimated based on aggregate population data.

In this paper, we develop an abstract approximation framework and convergence theory for formulating and solving just such an estimation problem. In addition to the theory, we have also included a number of examples and numerical results. However, we do not discuss here the application of these ideas to either the alcohol biosensor problem discussed above or even the deconvolution problem. Those results are presented elsewhere ([31, 32, 33]). In our treatment here, we are strictly concerned with the problem of estimating the distributions of random parameters in a forward model from a particular class of abstract linear infinite dimensional systems for which the input is known and observations of the output for a sampling of members of the target population are available. That is, we are referring to the problem of fitting the population model.

The class of systems we consider here are those governed by abstract parabolic or hyperbolic operators with damping formulated in a Gelfand triple setting together with input and observations on the boundary of the domain. These types of operators are sometimes referred to as being regularly dissipative, and can typically be shown to generate holomorphic or analytic semigroups. We formulate the estimation problem in much the same way as it is in standard linear regression. That is, that each data point is assumed to be an observation of the mean population behavior plus random error. We then formulate the estimation problem as an optimization problem over the space of feasible distributions for the random parameters. The objective of the optimization problem is to minimize prediction error in the form of the difference between the observed output signal and the expectation of the output of the model. We then consider a sequence of approximating estimation problems in each of which the infinite dimensional system is replaced by a finite dimensional approximating system. We then demonstrate that under appropriate (and readily verifiable) assumptions, the solutions to the approximating estimation problems converge to a solution to the original estimation problem with the infinite dimensional state. These convergence results are formulated in a functional analytic or operator theoretic setting and are based on ideas and results from linear semigroup theory.

Our general approach relies heavily on three relatively recent papers: 1) Banks and Thompson’s [7] framework for the estimation of probability measures in random abstract evolution equations and the convergence of finite dimensional approximations in the Prohorov metric, 2) a more recent and enhanced version of the previous paper, [2], and 3) Gittelson, Andreev, and Schwab’s [20] theory for random abstract parabolic partial differential equations with dynamics defined in terms of coercive sesquilinear forms. While our effort here is similar in spirit and takes its cue from the treatment in [2] and [7], it is somewhat different in that we are forced to assume that the probability measures that describe the distribution of our random parameters can be defined in terms of a joint density function; that is, that the random parameters are jointly absolutely continuous.

The approach in [20] is novel in the way that it treats the random parameters in the PDE as another space-like independent variable. This is done by appropriately defining corresponding Bochner spaces in which the weak formulation of the problem is stated and shown to be well-posed. In fact, it turns out that the random parameter dependent regularly dissipative operators that determine the underlying PDE are regularly dissipative when embedded in these Bochner spaces. Consequently, we are able to use linear semigroup theory to develop our approximation framework in much the same way as we have in our earlier deterministic treatments. In this way, finite dimensional approximation is handled in much the same way that it is for the standard deterministic space variables, and the estimation of the distribution of the random parameters effectively becomes analogous to the problem of estimating a variable coefficient in a deterministic PDE, a problem which has been studied extensively over the last thirty years ([4] and [6]).

We use the framework in [20] together with generation and approximation results from linear semigroup theory, (i.e. the Hille-Yosida-Phillips theorem and a version of the Trotter Kato approximation theorem) to establish that the sufficient conditions for a Banks Thompson-like convergence result are satisfied. These theoretical results allow us to develop rigorously established convergent computational algorithms that yield numerical approximations to the desired distributions. Moreover, the solutions in the Bochner spaces and their finite dimensional approximations directly capture the explicit dependence of the state and output (and eventually the deconvolved input) on the random parameters. Using this together with the estimated distributions for the random parameters, it becomes straight forward to directly identify credible intervals for the output without having to re-solve the PDE many times as you would if you were attempting to identify these credible intervals by naive sampling.

An outline of the remainder of the paper is as follows. In Section (2) we formally develop the estimation problem, reformulate it as a nonlinear least squares optimization problem and establish the existence of solutions. In Section (3) we discuss infinite dimensional systems described by regularly dissipative operators involving unbounded input and output (this is typically the case for a PDE with input and output on the boundary). In Section (4) we discuss the framework in [20] for treating systems of the form discussed in Section (3) but now involving random parameters. Our approximation and convergence results are presented in Section (5) and a discussion of examples and our numerical results are in Section (6). Section (7) has a few concluding remarks regarding where we plan to go next with this line of research.

In our discussions to follow we will on occasion use the notation E[X∥f], $\mathbb{E}[X\Vert F]$, or E [X∥π] to denote the expectation of the random variable X with respect to the probability density function f, the cumulative distribution function F, or the probability measure π. We use the “double bar” as opposed to a “single bar” to distinguish what we mean here with conditional expectation.

2. Estimation of Random Discrete Time Dynamical Systems

We consider the family of discrete or sampled time initial value problems that are set in an, in general, infinite dimensional Hilbert state space, $\mathcal{H}$, given by

$$x_{j+1,i}=g(t_j,x_{j,i},u_i;q),j=0,\dots ,n_i,i=1,2,\dots ,m,$$ (2.1)

$$x_{0,i}=x_{0,i}(q),i=1,2,\dots ,m,$$ (2.2)

where $g:{\mathbb{R}}^{+}\times \mathcal{H}\times {\prod }_{j=0}^{n_i}{\mathbb{R}}^{\mu }\times Q\to \mathcal{H}$ and for j = 0, …, n_i and i = 1, 2, …, m, u_i = {u_i,j} is an external input or control with $u_{i,j}\in {\mathbb{R}}^{\mu }$, and t_j = jτ, with τ > 0 the length of the sampling interval, describing the dynamics of a process common to the entire population. In addition, we assume that we can observe some function of the solutions of (2.1)-(2.2), x_j,i, as given by the output equation

$$y_{j,i}=y(t_j,x_{0,i},u_i;q)=C(x_{j,i},x_{0,i},u_i;q),j=0,\dots ,n_i,i=1,2,\dots ,m,$$ (2.3)

where $C:\mathcal{H}\times \mathcal{H}\times {\prod }_{j=0}^{n_i}{\mathbb{R}}^{\mu }\times Q\to {\mathbb{R}}^{\nu }$.

In equations (2.1)-(2.3), we assume q ∈ Q, where Q is the set of admissible parameters (a subset of Euclidean space endowed with Lebesgue measure), and the values of the parameters are specific to each individual in the population. Therefore, assuming that the parameters, q, are samples from a random vector $\mathcal{q}$, the objective is to estimate their (joint) distribution based on the aggregate data sampled from the population. For this purpose, we assume that the distribution of these random vectors is described by the joint pdf $f_0\in \mathcal{F}(Q)$, where $\mathcal{F}(Q)$ represents a set of feasible pdfs with support in Q.

There are a number of ways to formulate the statistical model that will be used as the basis for the estimation of the distribution of the random parameters. One approach is to treat (2.1)- (2.3) as an, in general, nonlinear mixed effects model (see, for example, [16, 17, 18, 32]) wherein randomness in the parameters, q, are used to quantify uncertainty between subjects, and randomness in the output or measurements, y_j,i given in (2.3) is intended to capture uncertainty within individual subjects. In this case we assume that the observed data points are of the form

$$V_{j,i}=y_{j,i}+{\epsilon }_{j,i},j=0,\dots ,n_i,i=1,\dots ,m,$$

where ε_j,i, j = 0, …, n_i, i = 1, …, m, representing measurement noise are assumed to be independent across subjects (i.e with repsect to i), conditionally independent with respect to $\mathcal{q}$ within subjects (i.e with repsect to j), identically distributed with mean 0 and known common variance σ², and with ε_j,i ~ φ, j = 0, …, n_i, i = 1, …, m. In this case, for example, using conditional probability and the total probability formula, a likelihood function could be defined formally as

$$\mathcal{L}(f_0;\{V_{j,i}\})=\prod _{i=1}^m{\int }_Q{L}_i(q;\{V_{j,i}\})f_0(q)dq=\prod _{i=1}^m{\int }_Q\prod _{j=0}^{n_i}\phi (V_{j,i}-C(x_{j,i},x_{0,i},u_i;q))f_0(q)dq.$$

Once one deals with a number of computational issues, specifically, the discretization or parameterization of f₀, finite dimensional approximation of the in general infinite dimensional state equation (2.1), the efficient evaluation of a potentially high dimensional integral, the loss of precision and underflow issues due to the fact that the evaluation of $\mathcal{L}$ requires the computation of products of small numbers, etc., one could then seek a maximum likelihood estimator for f₀ by maximizing $\mathcal{L}$ or, more typically, an expression involving $log\mathcal{L}(f_0;\{V_{j,i}\})$ to avoid having to deal with the products. Under appropriate regularity assumptions on φ, f₀, and the system (2.1)- (2.3), one way to do this might be via a gradient based search. Another might be via stochastic optimization. One could also treat direct observations of $\mathcal{q}$ as missing data and then use the iterative E-M algorithm to find the MLE (see, for example, [12]).

Alternatively, one could use the likelihood function defined above and take a Bayesian approach (see, for example, [8, 9, 10, 15, 33, 34]). One way of doing this would be to assume f₀ = f₀(·; ρ) has been parameterized by a parameter vector $\rho \in \mathcal{R}$, where $\mathcal{R}$ denotes a parameter set. Then assume a prior $\mathcal{P}$ on ρ and apply Bayes to obtain the posterior $\widehat{\mathcal{P}}$ as

$$\widehat{\mathcal{P}}(\rho )=\widehat{\mathcal{P}}(\rho \mid \{V_{j,i}\})=\frac{1}{Z}\widehat{\mathcal{L}}(\rho ;\{V_{j,i}\})\mathcal{P}(\rho )=\frac{1}{Z}\mathcal{L}(f_0(\cdot ;\rho );\{V_{j,i}\})\mathcal{P}(\rho ).$$

where Z is the normalizing constant given by

$$Z={\int }_{\mathcal{R}}\widehat{\mathcal{L}}(\rho ;\{V_{j,i}\})\mathcal{P}(\rho )d\rho ={\int }_{\mathcal{R}}\prod _{i=1}^m{\int }_Q\prod _{j=0}^{n_i}\phi (V_{j,i}-C(x_{j,i},x_{0,i},u_i;q))f_0(q;\rho )dq\mathcal{P}(\rho )d\rho .$$

Still another Bayesian approach could be used to estimate the distribution of $\mathcal{q}\sim f_0$ directly where now the posterior for $\mathcal{q},\widehat{\mathcal{P}}=\widehat{\mathcal{P}}(q)$ serves as the estimator for f₀. In this case we assume that ε_j,i, j = 0, …, n_i, i = 1, …, m are simply independent both across and within subjects, identically distributed with mean 0 and known common variance σ², and with ε_j,i ~ φ, j = 0, …, n_i, i = 1, …, m. If we now let $\mathcal{P}$ denote the prior for $\mathcal{q}$, then Bayes yields

$$\widehat{\mathcal{P}}(q)=\widehat{\mathcal{P}}(q\mid \{V_{j,i}\})=\frac{1}{Z}\prod _{i=1}^m{L}_i(q;\{V_{j,i}\})\mathcal{P}(q)=\frac{1}{Z}\prod _{i=1}^m\prod _{j=0}^{n_i}\phi (V_{j,i}-C(x_{j,i},x_{0,i},u_i;q))\mathcal{P}(q),$$

where the normalizing constant Z is now given by

$$Z={\int }_Q\prod _{i=1}^m{L}_i(q;\{V_{j,i}\})\mathcal{P}(q)dq={\int }_Q\prod _{i=1}^m\prod _{j=0}^{n_i}\phi (V_{j,i}-C(x_{j,i},x_{0,i},u_i;q))\mathcal{P}(q)dq.$$

Both of these Bayesian approaches also have some of the same computational issues as the MLE approach when some sort of MCMC technique such as Metropolis-Hastings or the Gibbs Sampler is used to sample the posterior distribution.

In our study here, however, we take a statistically somewhat less sophisticated approach. We consider the naive pooled data estimator. We do this for a number of reasons. 1) Our primary focus here is the finite dimensional approximation of the infinite dimensional state equation and the convergence of the corresponding estimators and the computational challenges described above would only serve to confound our findings, 2) The naive pooled estimator meshes especially well with the approach we take in dealing with the randomness in the family of PDEs (i.e. abstract parabolic, and eventually, damped hyperbolic) of particular interest to us here in the context of the alcohol biosensor problem described earlier. 3) A reasonable argument could be made that the data we observe is best described as pooled or averaged. We note that it in fact turns out that the approximation and convergence results we present here are highly relevant to the MLE and Bayesian approaches described in the previous paragraphs; we are currently investigating that and we will report on our findings and results in those cases elsewhere. Finally it is interesting to note that in the Bayesian approach, if the prior f₀ and the distribution of the measurement noise process, ε_j,i, as described by the density φ are both assumed to be normal, then the naive pooled data estimator we find here is in fact the Maximum A-Posteriori, or MAP, estimator.

In light of this, our statistical model assumes that the observed data points can be represented by the mean output of the model plus random error. Thus, we assume that we have random observations of the process given by a random array with components

$$V_{j,i}=\mathbb{E}[y_{j,i}\mid \mid f_0]+{\epsilon }_{j,i},j=0,\dots ,n_i,i=1,\dots ,m,$$ (2.4)

where in (2.4), ε_j,i, j = 0, …, n_i, i = 1, …, m, represent measurement noise and are assumed to be independent and identically distributed with mean 0 and known common variance σ². For $f\in \mathcal{F}(Q)$, define

$$v_i(t_j;f)=\mathbb{E}[y(t_j,x_{0,i},u_i;\mathcal{q})\mid \mid f]={\int }_{Q}C(x_{j,i},x_{0,i},u_i;q)f(q)dq,$$ (2.5)

the mean behavior at time t_j, j = 0, …, n_i, if $\mathcal{q}\sim f$.

The estimation problem is to estimate the pdf, f₀, using a least squares approach

$$\widehat{f}=\underset{f\in \mathcal{F}(Q)}{\arg \min }J(f;V)=\underset{f\in \mathcal{F}(Q)}{\arg \min }\sum _{i=1}^m\sum _{j=0}^{n_i}{(V_{j,i}-v_i(t_j;f))}^2.$$ (2.6)

where the v_i(t_j; f) are as given in (2.5).

Solving the optimization problem given in (2.6) will typically require finite dimensional approximation of the dynamical system given in (2.1)-(2.2), and the parameterization of the feasible set of pdfs, $\mathcal{F}(Q)$. Indeed, in our treatment here, we assume that the set of pdfs, $\mathcal{F}(Q)$, is parameterized by a vector of parameters θ ∈ Θ, where $\mathrm{\Theta }\subseteq {\mathbb{R}}^r$ is a set of feasible parameters. In this case, we denote the set of pdfs by ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$.

We approximate the estimation problem given in (2.6) by a sequence of finite dimensional estimation problems by replacing v_i(t_j; f) with a finite dimensional approximation $v_i^N(t_j;f)$. We obtain

$${\widehat{f}}^N=\underset{f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)}{\arg \min }{J}^N(f;V)=\underset{f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)}{\arg \min }\sum _{i=1}^m\sum _{j=0}^{n_i}{(V_{j,i}-v_i^N(t_j;f))}^2.$$ (2.7)

We note that ultimately, we will want to dispense with the assumption that $\mathcal{F}(Q)$ has been parametrized by the finite dimensional parameter θ ∈ Θ and actually estimate the shape of f directly. In this case, $\mathcal{F}(Q)$ will also have to be approximated or discretized with the level, or dimension of the parameterization having to grow in order to establish convergence. We are currently studying this extension to the results presented here and will discuss our findings elsewhere. Analogous to theorem 5.1 in [7], we have the following convergence result for the ${\widehat{f}}^{N,}\mathrm{s}$.

Theorem 2.1. Let $\mathrm{\Theta }\subseteq {\mathbb{R}}^r$ be compact. If

A. The maps on Θ, θ ↦ f (q; θ), for almost every q ∈ Q, and θ ↦ J^N (f (·; θ); V), for all N and $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$ are continuous,

B. For any sequence of densities $f_N\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$ with lim_N→∞ f_N(q) = f (q), a.e. q ∈ Q, for some $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$, we have $v_i^N(t_j;f_N)$ converging to v_i(t_j; f) for all i ∈ {1, …, m} and j ∈ {0, …, n_i} as N → ∞, and

C. The v_i(t_j; f) and $v_i^N(t_j;f)$ are uniformly bounded for all j ∈ {0, …, n_i}, i ∈ {1, …, m} and $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$,

then it will follow that there exist solutions ${\widehat{f}}^N$ to the estimation problems over ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$, given in (2.7), and there exists a subsequence of the ${\widehat{f}}^{N,}s$ that converges to a solution $\widehat{f}$ of the estimation problem over ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$ given in (2.6).

Proof. Finding the solution to the problem in (2.7) is equivalent to finding the parameters θ ∈ Θ such that J^N (f; V) is minimized. Since Θ is a compact set and the map θ → J^N (f(·; θ); V) is continuous for all N by (A), a solution ${\widehat{f}}^N$ to the estimation problem (2.7) over ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$ exists.

Next, let $\{f_N\}\subseteq {\mathcal{F}}_{\mathrm{\Theta }}(Q)$ be any sequence with lim_N→∞ f_N(q) = f (q), a.e. q ∈ Q for some $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$ and consider that

$$\begin{gathered} \mid J^N(f_N;V)-J(f;V)\mid =\mid \sum _{i=1}^m\sum _{j=0}^{n_i}{(V_{j,i}-v_i^N(t_j;f_N))}^2 \\ -\sum _{i=1}^m\sum _{j=0}^{n_i}{(V_{j,i}-v_i(t_j;f))}^2\mid \\ \le \sum _{i=1}^m\sum _{j=0}^{n_i}\mid 2V_{j,i}-(v_i(t_j;f)+v_i^N(t_j;f_N))\mid \\ \cdot \mid v_i(t_j;f)-v_i^N(t_j;f_N)\mid \\ \le M\sum _{i=1}^m\sum _{j=0}^{n_i}\mid v_i(t_j;f)-v_i^N(t_j;f_N)\mid , \end{gathered}$$

for some M > 0, since v_i(t_j; f) and $v_i^N(t_j;f)$ are uniformly bounded for all i ∈ {1, …, m} and j ∈ {0, …, n_i} (by assumption (C)), and $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$. Then, by (B), we obtain

$$J^N(f_N;V)\to J(f;V),$$ (2.8)

as N → ∞. On the other hand, since ${\widehat{f}}^N=\widehat{f}(\cdot ;{\widehat{\theta }}^N)$, where ${\widehat{\theta }}^N\in \mathrm{\Theta }$, is the minimizer of J^N(f; V), we have

$$J^N({\widehat{f}}^N;V)\le J^N(f;V),$$ (2.9)

for all $f=f(\cdot ;\theta )\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$ and N = 1, 2, …. Since $\{{\widehat{\theta }}^N\}\subset \mathrm{\Theta }$, compact, there exists a subsequence ${\widehat{\theta }}^{N_k}$ with ${\widehat{\theta }}^{N_k}\to \widehat{\theta }$ as k → ∞. Thus, taking the limit as k → ∞ in (2.9) with N replaced by N_k, and using (2.8) (with $f_k^N=f$, all k =1, 2, … when the limit is taken on the right hand side of (2.9)), we obtain

$$J(\widehat{f};V)\le J(f;V),$$ (2.10)

for all $f\in {\mathcal{F}}_{\mathrm{\Theta }}(Q)$, where $\widehat{f}=\widehat{f}(\cdot ;\widehat{\theta })$. Thus, (2.10) implies that $\widehat{f}$ is a solution of estimation problem given in (2.6) over ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$. □

3. Abstract Parabolic Systems with Unbounded Input and Output

Let V and H be in general complex (but in many instances, real would suffice) Hilbert spaces with V ↪ H, i.e. V is continuously and densely embedded in H. By identifying H with its dual H*, we obtain the Gelfand triple V ↪ H ↪ V*. Let < ·, · >_H denote the H inner product and ∣·∣_H, ∥·∥_V denote norms on H and V, respectively, and assume that (Q, d_Q) is a compact metric space contained in Euclidean space endowed with Lebesgue measure. In what follows all multi-dimensional vectors, whether in Euclidean or some abstract space, are assumed to be column vectors, unless explicitly stated otherwise. For q ∈ Q, let $a(q;\cdot ,\cdot ):V\times V\to \mathbb{C}$ be a sesquilinear form that has the following properties

1. Boundedness There exists a constant α₀ > 0 such that ∣a(q; ψ₁, ψ₂)∣ ≤ α₀∥ψ1∥_V∥ψ₂∥_V, ψ₁, ψ₂ ∈ V, q ∈ Q,

2. Coercivity There exist constants ${\lambda }_0\in \mathbb{R}$ and μ₀ > 0 such that $a(q_1;\psi ,\psi )+{\lambda }_0\mid \psi {\mid }_H^2\ge {\mu }_0\Vert \psi {\Vert }_V^2$, ψ ∈ V, q ∈ Q,

3. Measurability For all ψ₁, ψ₂ ∈ V, the map q ↦ a(q; ψ₁, ψ₂) is measurable on Q with respect to all measures defined in terms of the densities in ${\mathcal{F}}_{\mathrm{\Theta }}(Q)$, where $\mathrm{\Theta }\subseteq {\mathbb{R}}^r$ is the set of feasible parameters.

Assume further that b(q), c(q) are respectively μ and ν dimensional row vectors in V* with the maps q ↦< b(q), ψ >_V*,V and q ↦< c(q), ψ >_{V*, V} measurable on Q for ψ ∈ V, where < ·, · >_V*,V denotes the duality pairing between V and V*. We consider the system which is written in weak form as

$$\begin{gathered} {\langle \stackrel{.}{x},\psi \rangle }_{V^{\ast },V}+a(q;x,\psi )={\langle b(q),\psi \rangle }_{V^{\ast },V}u,\psi \in V, \\ x(0)=x_0\in H, \\ y(t)={\int }_0^T{\langle c(q),x_{(t)}(s)\rangle }_{V^{\ast },V}ds, \end{gathered}$$ (3.1)

where T > 0, and φ_(t)(s) = φ(t − s)χ_{[0, T]}(s), s ∈ [0, T]. For $u\in L_2([0,T],{\mathbb{R}}^{\mu })$, it can be shown that (3.1) has a unique solution (see [24, 39]) $x\in W(0,T)≔\{\psi :\psi \in L_2([0,T],V),\stackrel{.}{\psi }\in L_2([0,T],V^{\ast })\}\subseteq C([0,T],H)$ which depends continuously on $u\in L_2([0,T],{\mathbb{R}}^{\mu })$. It follows that $y\in L_2([0,T],{\mathbb{R}}^{\nu })$.

For q ∈ Q, under the assumptions (i),(ii), the sesquilinear form a(q; ·, ·) defines a bounded linear operator A(q) : V → V* by < A(q)ψ₁, ψ₂ >_V*,V = −a(q; ψ₁, ψ₂) where ψ₁, ψ₂ ∈ V. It can be shown further that (see [3, 5, 39]) A(q) restricted to the set Dom(A(q)) = {ϕ ∈ V : A(q)ϕ ∈ H} is the infinitesimal generator of a holomorphic or analytic semigroup of bounded linear operators on H. Moreover, this semigroup can be restricted to be a holomorphic semigroup on V and extended to a holomorphic semigroup on V* by appropriately restricting or extending the domain, Dom(A(q)), of the operator A(q) (see, for example, [3] and [39]).

For q ∈ Q, define the operators $B(q):{\mathbb{R}}^{\mu }\to V^{\ast }\text{by}\langle B(q)u,\phi {\rangle }_{V^{\ast },V}=\langle b(q),\phi {\rangle }_{V^{\ast },V}u$ and $C(q):L_2([0,T],V)\to {\mathbb{R}}^{\nu }$ by $C(q)\psi ={\int }_0^T\langle c(q),\psi (s){\rangle }_{V^{\ast },V}ds$, for $u\in {\mathbb{R}}^{\mu }$, φ ∈ V, and ψ ∈ L₂([0, T], V), and rewrite the system in (3.1) as

$$\begin{gathered} \stackrel{.}{x}(t)=A(q)x(t)+B(q)u(t), \\ x(0)=x_0, \\ y(t)=C(q)x_{(t)},t>0. \end{gathered}$$ (3.2)

The mild solution of (3.2) is given by the variation of constants formula as

$$x(t;q)=e^{A(q)t}{x}_0+{\int }_0^t{e}^{A(q)(t-s)}B(q)u(s)ds,t\ge 0.$$ (3.3)

Moreover, since the semigroup {e^A(q)t : t ≥ 0} is analytic it follows that

$$y(t;q)=C(q)x_{(t)}(q)={\int }_0^T{\langle c(q),x_{(t)}(s;q)\rangle }_{V^{\ast },V}ds,t\ge 0.$$ (3.4)

is well defined.

3.1. The Discrete Time Formulation

Now let τ > 0 be a sampling time and consider zero-order hold inputs of the form u(t) = u_j, t ∈ [jτ,(j + 1)τ), j = 0, 1, 2, …. Setting x_j = x(jT), for j = 0, 1, 2, …, (3.3) and (3.4) yield that

$$x_{j+1}=\widehat{A}(q)x_j+\widehat{B}(q)u_j,y_i=\widehat{C}(q)x_{(j)},j=0,1,2,\dots$$ (3.5)

where now we let x₀ ∈ V. Here, again by the properties of the analytic semigroup (see [26, 39]), we have {e^A(q)t : t ≥ 0}, x_j ∈ V, $\widehat{A}(q)=e^{A(q)\tau }\in \mathcal{L}(V,V)$ and $\widehat{B}(q)={\int }_0^{\tau }{e}^{A(q)s}B(q)ds\in \mathcal{L}({\mathbb{R}}^{\mu },H)$. The operator $\widehat{C}(q)$ appearing in (3.5) is defined by recalling (3.4). We set

$$\widehat{C}(q)x_{(j)}=C(q)x_{(j)},$$ (3.6)

where x_(j) in (3.6) denotes the function in L₂(0, T, V) given by

$$x_{(j)}=\sum _{i=1}^j{x}_i{\chi }_{[(j-1)\tau ,j\tau ]}.$$ (3.7)

Now, in light of the coercivity assumption, Assumption (ii), by making the change of variables z(t) = e^−λ₀tx(t) and v(t) = e^−λ₀tu(t), without loss of generality we may assume that the operator A(q) is invertible with bounded inverse. Thus we have that ${\phantom{\mid }\widehat{B}(q)={\int }_0^{\tau }{e}^{A(q)s}B(q)ds=A(q{)}^{-1}{e}^{A(q)s}B(q)\mid }_0^{\tau }=(\widehat{A}(q)-I)A(q{)}^{-1}B(q)\in \mathcal{L}({\mathbb{R}}^{\mu },V)$. It follows that the recurrence given in (3.5) is a recurrence in V with $\widehat{A}(q)\in \mathcal{L}(V,V)$ and $\widehat{B}(q)\in \mathcal{L}({\mathbb{R}}^{\mu },V)$. Thus it now becomes possible to allow the discrete time output operator $\widehat{C}(q)\in \mathcal{L}(V,{\mathbb{R}}^{\nu })$ defined in (3.6) and (3.7), if so desired, to take on the much simpler form $\widehat{C}(q)x=\langle c(q),x{\rangle }_{V^{\ast },V}$. In what follows we shall assume that the output operator takes this simpler form.

3.2. Systems with Boundary Input

Of primary interest to us here are systems of the form (3.1) or (3.2) where the input u is on the boundary of the spatial domain. The theory developed in [13] and [27] tells us how in this case to define the input operator B(q) and the notion of a mild solution upon which our approach is based. Let W be a Hilbert space which is densely and continuously embedded in H. Let $\Delta (q)\in \mathcal{L}(W,H)$ and $\mathrm{\Gamma }(q)\in \mathcal{L}(W,{\mathbb{R}}^{\mu })$ and assume that $Dom(A(q))\subseteq \mathcal{N}(\mathrm{\Gamma }(q))\subseteq W$, Γ(q) is surjective and Δ(q) = A(q) on Dom(A(q)). We then consider the system with input on the boundary given by

$$\begin{gathered} \stackrel{.}{x}(t)=\mathrm{\Delta }(q)x(t),t>0, \\ \mathrm{\Gamma }(q)x(t)=u(t),t>0, \\ y(t)=C(q)x_{(t)},t>0, \\ x(0)=x_0. \end{gathered}$$ (3.8)

In [13], Curtain and Salamon define a solution to the system (3.8) for the case where $u\in C([0,T];{\mathbb{R}}^{\mu })$ and x₀ ∈ W with Γ(q)x₀ = u(0), to be a function x ∈ C([0, T]; W) ∩ C¹([0, T]; H) that satisfies (3.8) at every t ∈ (0, T). The operator A(q) densely defined implies that it has an adjoint operator A(q)* : Dom(A(q)*) ⊆ H → H which is also densely defined and closed. Defining Z* to be the Hilbert space Dom(A(q)*) endowed with the graph Hilbert space norm associated with A(q)*, Z* will be continuously and densely embedded in H. So, the Gelfand triple Z* ↪ H ↪ Z is obtained where Z = Z** represents the dual space of Z*. By definition $A(q{)}^{\ast }\in \mathcal{L}(Z^{\ast },H)$ and consequently therefore, $A(q)\in \mathcal{L}(H,Z)$. It follows that the semigroup {e^A(q)t : t ≥ 0} can be uniquely extended to a holomorphic semigroup on Z with infinitesimal generator A(q) : H ⊆ Z → Z, the extension A(q) to H defined via the duality pairing < A(q)ψ, ϕ>_Z,Z*=< ψ, A(q)* ϕ >_H, for ψ ∈ H, and ϕ ∈ Z* = Dom(A(q)*).

For each q ∈ Q, let ${\mathrm{\Gamma }}^{+}(q)\in \mathcal{L}({\mathbb{R}}^{\mu },W)$ be any right inverse of $\mathrm{\Gamma }(q)\in \mathcal{L}(W,{\mathbb{R}}^{\mu })$, and define the operator $B(q)\in \mathcal{L}({\mathbb{R}}^{\mu },Z)$ by B(q) = (Δ(q) – A(q))Γ+(q). It is not difficult to show that B(q) is well defined (i.e. that it does not depend on the particular choice of the right inverse Γ⁺ (q)). Then for any x₀ ∈ H and $u\in L_2([0,T];{\mathbb{R}}^{\mu })$, the mild solution, x ∈ C([0, T]; Z), of the initial boundary value problem in (3.8) is the Z-valued function given by

$$x(t)=e^{A(q)t}{x}_0+{\int }_0^t{e}^{A(q)(t-s)}B(q)u(s)ds,t\ge 0.$$ (3.9)

It is shown in [13] that if (3.8) has a solution, then it is given by (3.9) where x ∈ C([0, T], H) ∩ H ¹ ((0, T), Z) and moreover, we have that the estimate given by $\mid {\int }_0^t{e}^{A(q)(t-s)}B(q)u(s)ds{\mid }_H\le k\Vert u{\Vert }_{L_2([0,T];{\mathbb{R}}^{\mu })}$ holds.

We note that if in fact we have that W ⊂ V, which is often the case (for example, in a one dimensional diffusion equation with either Neumann or Robin boundary input (see our examples in Section (6) below), but may not be the case if, for example, the boundary input is Dirichlet), then in the above formulation we may take Z* = V and Z = V*. In this case it will follow that $B(q)=(\Delta (q)-A(q)){\mathrm{\Gamma }}^{+}(q)\in \mathcal{L}({\mathbb{R}}^{\mu },V^{\ast })$ and consequently that the theory presented at the beginning of Section (3), and in particular, the discrete time theory presented in Section (3.1), applies. For ease of exposition, we will assume that this is indeed the case for what follows below. We note that all the results continue to follow in the more general case where Z* = Dom(A(q)*). It then follows that $\widehat{A}(q)=e^{A(q)\tau }\in \mathcal{L}(V,V)$ and that $\widehat{B}(q)={\int }_0^{\tau }{e}^{A(q)s}dsB(q)\in \mathcal{L}({\mathbb{R}}^{\mu },V)$ and therefore that

$$\widehat{B}(q)={\phantom{\mid }{\int }_0^{\tau }{e}^{A(q)s}B(q)ds=A{(q)}^{-1}{e}^{A(q)s}B(q)\mid }_0^{\tau }=(\widehat{A}(q)-I)A{(q)}^{-1}B(q),$$

and $\widehat{C}(q)=C(q)\in \mathcal{L}(V,{\mathbb{R}}^{\nu })$. Note that now we have

$$\widehat{B}(q)=(I-\widehat{A}(q)){\mathrm{\Gamma }}^{+}(q)+{\int }_0^{\tau }{e}^{A(q)s}ds\mathrm{\Delta }(q){\mathrm{\Gamma }}^{+}(q)\in \mathcal{L}({\mathbb{R}}^{\mu },V),$$ (3.10)

and if Γ⁺(q) can be chosen so that $R({\mathrm{\Gamma }}^{+}(q))\in \mathcal{N}(\Delta (q))$, then the expression in (3.10) becomes $\widehat{B}(q)=(I-\widehat{A}(q)){\mathrm{\Gamma }}^{+}(q)$. Then, if x₀ = 0 ∈ H, y_i is given by

$$\begin{gathered} y_i=\sum _{j=0}^{i-1}C(q)\widehat{A}{(q)}^{i-j-1}\widehat{B}(q)u_i \\ =\sum _{j=0}^{i-1}{K}_{i,j}{u}_j,i=1,2,\dots , \end{gathered}$$ (3.11)

where the operator $K_{i,j}=C(q)\widehat{A}(q{)}^{i-j-1}(I-\widehat{A}(q)){\mathrm{\Gamma }}^{+}(q)$ appearing in (3.11) is the gain that represents the contribution of the j^th input channel to the i^th output channel.

4. Random Regularly Dissipative Operators and Their Associated Semigroups

In this section, we summarize the key ideas from the framework developed in [20] and [30] which are central to our approach. We assume that $\mathcal{q}$ is a p-dimensional random vector whose support is in ${\prod }_{i=1}^p[a_i,b_i]$ where $-\infty <\stackrel{‒}{\alpha }<a_i<b_i<\stackrel{‒}{\beta }<\infty$ for all i = 1, 2, …, p. Letting $\overrightarrow{a}=[a_i{]}_{i=1}^p$, $\overrightarrow{b}=[b_i{]}_{i=1}^p$ and let $\mathrm{\Theta }\subset {\mathbb{R}}^r$ for some r be closed and bounded. We assume that the distribution of $\mathcal{q}$ can be represented by an absolutely continuous cumulative distribution function $F(q;\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$, or equivalently, by a (push forward) measure $\pi =\pi (\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$, where $\overrightarrow{\theta }\in \mathrm{\Theta }$. Let a(·; ·, ·) be a sesquilinear form satisfying (i)-(iii) given in Section (3), where the assumed measurability is with respect to all of the measures $\pi =\pi (\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$.

Define the Bochner spaces $\mathcal{V}=L_{\pi }^2(Q;V)$ and $\mathcal{H}=L_{\pi }^2(Q;H)$. The assumptions from Section (3) on the spaces V and H guarantee that the spaces $\mathcal{V}$, $\mathcal{H}$ and ${\mathcal{V}}^{\ast }$ form the Gelfand triple $\mathcal{V}\hookrightarrow \mathcal{H}\hookrightarrow {\mathcal{V}}^{\ast }$ (see [20]) where $\mathcal{H}$ is identified with its dual ${\mathcal{H}}^{\ast }$ and ${\mathcal{V}}^{\ast }$ is identified with $L_{\pi }^2(Q;V^{\ast })$.

For $\overrightarrow{a}=[a_i{]}_{i=1}^p$, $\overrightarrow{b}=[b_i{]}_{i=1}^p$ satisfying $-\infty <\stackrel{‒}{\alpha }<a_i<b_i<\stackrel{‒}{\beta }<\infty$ for all i = 1, 2, …, p, and $\overrightarrow{\theta }\in \mathrm{\Theta }$, set $\rho =(\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$. Then we define the π(ρ)-averaged sesquilinear forms $\mathcal{a}(\rho ;\cdot ,\cdot ):\mathcal{V}\times \mathcal{V}\to \mathbb{C}$ (note, the spaces $\mathcal{H}$, $\mathcal{V}$, and ${\mathcal{V}}^{\ast }$ now of course depend on ρ, but our notation here we will not explicitly show this dependence unless clarity demands it) by

$$\mathcal{a}(\rho ;\phi ,\psi )={\int }_{Q}a(q;\phi (q),\psi (q))d\pi (q;\rho )=\mathbb{E}[a(\mathcal{q};\phi (\mathcal{q}),\psi (\mathcal{q}))\mid \mid \pi (\rho )],$$ (4.1)

where $\phi ,\psi \in \mathcal{V}$ and $\rho =(\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$. It is not difficult to show that Assumptions (i)-(iii) imply that $\mathcal{a}(\rho ;\cdot ,\cdot )$ is a bounded and coercive sesquilinear form on $\mathcal{V}\times \mathcal{V}$. Consequently, this sesquilinear form defines a bounded linear map $\mathcal{A}(\rho ):\mathcal{V}\to {\mathcal{V}}^{\ast }$ by $<\mathcal{A}(\rho )\phi ,\psi {>}_{{\mathcal{V}}^{\ast },\mathcal{V}}=-\mathcal{a}(\rho ;\phi ,\psi )$ which when appropriately restricted or extended is the infinitesimal generator of analytic semigroups of bounded linear operators $\{e^{\mathcal{A}(\rho )t}:t\ge 0\}$ on $\mathcal{V}$, $\mathcal{H}$ and ${\mathcal{V}}^{\ast }$ (see [3, 5, 39]). We assume that the maps q ↦< b(q), ψ(q) >_V*,V and q ↦< c(q), ψ(q) >_V*,V are π(ρ)-measurable for any $\psi \in \mathcal{V}$, and that ∥b(q)∥_V*, ∥c(q)∥_V* are uniformly bounded for a.e. q ∈ Q. We then define $\mathcal{B}(\rho ):{\mathbb{R}}^{\mu }\to {\mathcal{V}}^{\ast }$ and $\mathcal{C}(\rho ):\mathcal{V}\to {\mathbb{R}}^{\nu }$ by

$$\begin{gathered} <\mathcal{B}(\rho )u,\psi {>}_{{\mathcal{V}}^{\ast },\mathcal{V}}={\int }_Q{\langle b(q),\psi (q)\rangle }_{V^{\ast },V}d\pi (q;\rho )u \\ =\mathbb{E}[{\langle b(\mathcal{q}),\psi (\mathcal{q})\rangle }_{V^{\ast },V}\mid \mid \pi (\rho )]u, \end{gathered}$$ (4.2)

$$\mathcal{C}(\rho )\psi ={\int }_Q{\langle c(q),\psi \rangle }_{V^{\ast },V}d\pi (q;\rho )=\mathbb{E}[{\langle c(\mathcal{q}),\psi (\mathcal{q})\rangle }_{V^{\ast },V}\mid \mid \pi (\rho )],$$ (4.3)

for $u\in {\mathbb{R}}^{\mu }$ and $\psi \in \mathcal{V}$.

With the definitions (4.1) - (4.3) of the operators $\mathcal{A}$, $\mathcal{B}$, and $\mathcal{C}$, consider the abstract evolution system given by

$$\begin{gathered} \stackrel{.}{x}(t)=\mathcal{A}(\rho )x(t)+\mathcal{B}(\rho )u(t), \\ x(0)=x_0\in H, \\ \mathcal{Y}(t)=\mathcal{C}(\rho )x(t),t>0, \end{gathered}$$ (4.4)

whose mild solution is given by

$$x(t)=\mathcal{T}(t;\rho )x_0+{\int }_0^t\mathcal{T}(t-s;\rho )\mathcal{B}(\rho )u(s)ds,t\ge 0,$$ (4.5)

where $\mathcal{T}(t;\rho )=\{e^{\mathcal{A}(\rho )t}:t\ge 0\}$ is the analytic semigroup generated by the operator $\mathcal{A}(\rho )$. From (4.4) and (4.5), it follows that

$$\mathcal{Y}(t)={\int }_0^t\mathcal{C}(\rho )\mathcal{T}(t-s;\rho )\mathcal{B}(\rho )u(s)ds,t\ge 0.$$ (4.6)

As in Section (3), we obtain a discrete or sampled time version of (4.4). Now let x₀ ∈ V, let τ > 0 be the sampling time, and consider zero-order hold inputs of the form u(t) = u_j, t ∈ [jτ, (j + 1)τ),τ), j = 0, 1, 2, …. Setting ${\mathcal{X}}_j=\mathcal{X}(j\tau )$ and ${\mathcal{Y}}_j=\mathcal{Y}(j\tau )$, j = 0, 1, 2, …, (4.5) and (4.6) yield

$$x_{j+1}=\widehat{\mathcal{A}}(\rho )x_j+\widehat{\mathcal{B}}(\rho )u_j,{\mathcal{Y}}_j=\widehat{\mathcal{C}}(\rho )x_j,j=0,1,2,\dots ,$$ (4.7)

with ${\mathcal{X}}_0\in \mathcal{V}$ and $\widehat{\mathcal{A}}(\rho )=\mathcal{T}(\tau ;\rho )\in \mathcal{L}(\mathcal{V},\mathcal{V})$, $\widehat{\mathcal{B}}(\rho )={\int }_0^{\tau }\mathcal{T}(s;\rho )\mathcal{B}(\rho )ds\in \mathcal{L}({\mathbb{R}}^{\mu },\mathcal{V})$, and $\widehat{\mathcal{C}}(\rho )=\mathcal{C}(\rho )\in \mathcal{L}(\mathcal{V},{\mathbb{R}}^{\nu })$. Note that the operators $\widehat{\mathcal{A}}(\rho )$ and $\widehat{\mathcal{B}}(\rho )$ are bounded since $\{\mathcal{T}(t;\rho ):t\ge 0\}$ is an analytic semigroup on $\mathcal{V}$, $\mathcal{H}$, and ${\mathcal{V}}^{\ast }$ (see [3, 5, 24, 39]). If $\mathcal{A}(\rho ):\mathit{Dom}(\mathcal{A}(\rho ))\subseteq {\mathcal{V}}^{\ast }\to {\mathcal{V}}^{\ast }$ has bounded inverse, then $\widehat{\mathcal{B}}(\rho )={\int }_0^{\tau }\mathcal{T}(s;\rho )\mathcal{B}(\rho )ds=\widehat{A}{(\rho )}^{-1}\mathcal{T}(s;\rho )\mathcal{B}(\rho ){\mid }_0^{\tau }=(\widehat{\mathcal{A}}(\rho )-I)\mathcal{A}{(\rho )}^{-1}\mathcal{B}(\rho )\in \mathcal{L}({\mathbb{R}}^{\mu },\mathcal{V})$.

It is shown in [20] and [30] that the solutions of systems (4.4) and (3.2) and (4.7) and (3.5) agree for π-a.e. q ∈ Q. It follows that

$$\mathcal{Y}(t)=\mathcal{C}(\rho )x(t)=\mathbb{E}[y(t;\mathcal{q})\mid \mid \pi (\rho )]=\mathbb{E}[C(\mathcal{q})x(t;\mathcal{q})\mid \mid \pi (\rho )],\forall t\ge 0,$$ (4.8)

and hence, from (4.8), that

$${\mathcal{Y}}_j=\widehat{\mathcal{C}}(\rho )x_j=\mathbb{E}[{\mathcal{Y}}_j(\mathcal{q})\mid \mid \pi (\rho )]=\mathbb{E}[\widehat{C}(\mathcal{q})x_j(\mathcal{q})\mid \mid \pi (\rho )],$$ (4.9)

where in (4.8) and (4.9) $\mathbb{E}[\cdot \Vert \pi ]$ denotes expectation with respect to the measure π.

5. Approximation and Convergence

In this section, we can now formally state our estimation problem and the sequence of finite dimensional approximating problems. We will also state and prove a convergence theorem.

5.1. The Estimation Problem

Assume that data of the form ${({\{{\tilde{u}}_{i,j}\}}_{j=0}^{n_i-1},{\{{\tilde{y}}_{i,j}\}}_{j=0}^{n_i})}_{i=1}^m$, has been given. Determine ${\rho }^{\ast }=({\overrightarrow{a}}^{\ast },{\overrightarrow{b}}^{\ast },{\overrightarrow{\theta }}^{\ast })\in \mathrm{\Xi },\mathrm{\Xi }$ a compact subset of ${\mathbb{R}}^{2p}\times \Theta \subset {\mathbb{R}}^{2p+r}$, ${\overrightarrow{a}}^{\ast }={[a_i^{\ast }]}_{i=1}^p$, ${\overrightarrow{b}}^{\ast }={[b_i^{\ast }]}_{i=1}^p$, which minimizes

$$J(\rho )=\sum _{i=1}^m{J}_i(\rho )=\sum _{i=1}^m\sum _{j=0}^{n_i}{\mid {\mathcal{Y}}_{i,j}({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},\rho )-{\tilde{y}}_{i,j}\mid }^2$$ (5.1)

where for i = 1, 2, …, m, ${\mathcal{Y}}_{i,j}({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},\rho )$ is given by (4.7) with $u_j={\tilde{u}}_{i,j}$, j = 0, …, n, i = 1, 2, …, m, and (4.9).

Recalling the assumption that for i ∈ {1, 2, …, p}, $-\infty <\stackrel{‒}{\alpha }\le a_i<b_i\le \stackrel{‒}{\beta }<\infty$, let $\stackrel{‒}{Q}={\prod }_{i=1}^p[\stackrel{‒}{\alpha },\stackrel{‒}{\beta }]$. Let $\stackrel{‒}{\rho }=({[\stackrel{‒}{\alpha }]}_{i=1}^p,{[\stackrel{‒}{\beta }]}_{i=1}^p\overrightarrow{\theta })\in \Xi$,$\stackrel{‒}{\mathcal{H}}=L_{\pi (\stackrel{‒}{\rho })}^2(\stackrel{‒}{Q};H)$ and $\stackrel{‒}{\mathcal{V}}=L_{\pi \left(\stackrel{‒}{\rho }\right)}^2(\stackrel{‒}{Q};V)$. Then, for N = 1, 2, …, let ${\overrightarrow{a}}^N={[a_i^N]}_{i=1}^p$, ${\overrightarrow{b}}^N={[b_i^N]}_{i=1}^p$ be such that $-\infty <\stackrel{‒}{\alpha }\le a_i^N<b_i^N\le \stackrel{‒}{\beta }<\infty$, and let ${\rho }^N=([{\overrightarrow{a}}^N,{\overrightarrow{b}}^N,\overrightarrow{\theta })\in \Xi$. Set $Q^N={\prod }_{i=1}^p[a_i^N,b_i^N]$, ${\mathcal{H}}^N=L_{\pi ({\stackrel{‒}{\rho }}^N)}^2(Q^N,H)$, ${\mathcal{V}}^N=L_{\pi ({\stackrel{‒}{\rho }}^N)}^2(Q^N,V)$ and let ${\mathcal{U}}^N$ be a finite dimensional subspace of ${\mathcal{V}}^N$. Let ${\mathcal{L}}^N:\stackrel{‒}{\mathcal{H}}\to {\mathcal{H}}^N$ be a linear map defined by ${\mathcal{L}}^N(\psi )=\psi {\mid }_{Q^N}$ for any $\psi \in \stackrel{‒}{\mathcal{H}}$, let ${\mathcal{P}}^N:{\mathcal{H}}^N\to {\mathcal{U}}^N$ denote the orthogonal projection of ${\mathcal{H}}^N$ onto ${\mathcal{U}}^N$, and define ${\mathcal{J}}^N:\stackrel{‒}{\mathcal{H}}\to {\mathcal{U}}^N$ by ${\mathcal{J}}^N={\mathcal{P}}^N\circ {\mathcal{I}}^N$.

In addition, recall that we have assumed that for p ∈ Ξ, the probability distributions described by π(ρ) are all absolutely continuous; that is π(ρ) ~ f(ρ), where f(ρ) = f(·; ρ) is a joint density for the random vector $\mathcal{q}$.

Noting that in this formulation, ${\mathcal{U}}^N$ is neither a subspace of $\stackrel{‒}{\mathcal{H}}$ nor $\stackrel{‒}{\mathcal{V}}$, we define the operators ${\mathcal{A}}^N(\rho )$ on ${\mathcal{U}}^N$ to be what are essentially the restrictions of $\mathcal{A}(\rho )$ to the spaces ${\mathcal{U}}^N$. More precisely, we set

$$\begin{gathered} \langle {\mathcal{A}}^N(\rho )v^N,w^N\rangle =-\mathcal{a}(\rho ;v^N,w^N)=-{\int }_{Q}a(q;v^N(q),w^N(q))d\pi (q,\rho ) \\ =-{\int }_{Q}a(q;v^N(q),w^N(q))f(q;\rho )dq=-\mathbb{E}[a(\mathcal{q};v^N(\mathcal{q}),w^N(\mathcal{q}))\mid \mid \pi (\rho )], \end{gathered}$$ (5.2)

where ${\mathcal{V}}^N$, $w^N\in {\mathcal{U}}^N$.

Define the operators ${\mathcal{B}}^N(\rho ):{\mathbb{R}}^{\mu }\to {\mathcal{U}}^N$ and ${\mathcal{C}}^N(\rho ):{\mathcal{U}}^N\to {\mathbb{R}}^{\nu }$ by

$$\begin{gathered} <{\mathcal{B}}^N(\rho )u,v^N{>}_{{\mathcal{V}}^{\ast },\mathcal{V}}={\int }_Q{\langle b(q),v^N(q)\rangle }_{V^{\ast },V}d\pi (q;\rho )u \\ =\mathbb{E}[{\langle b(\mathcal{q}),v^N(\mathcal{q})\rangle }_{V^{\ast },V}\mid \mid \pi (\rho )]u, \end{gathered}$$ (5.3)

$${\mathcal{C}}^N(\rho )v^N={\int }_Q{\langle c(q),v^N\rangle }_{V^{\ast },V}d\pi (q;\rho )=\mathbb{E}[{\langle c(\mathcal{q}),v^N(\mathcal{q})\rangle }_{V^{\ast },V}\mid \mid \pi (\rho )],$$ (5.4)

where $v^N\in {\mathcal{U}}^N$, and $u\in {\mathbb{R}}^{\mu }$.

With these definitions, we can now state the finite dimensional approximating problems.

Assume that data of the form ${({\{{\tilde{u}}_{i,j}\}}_{j=0}^{n_i-1},{\{{\tilde{y}}_{i,j}\}}_{j=0}^{n_i})}_{i=1}^m$, has been given. Determine ${\rho }^{N\ast }=({\overrightarrow{a}}^{N\ast },{\overrightarrow{b}}^{N\ast },{\overrightarrow{\theta }}^{N\ast })\in \Xi$, Ξ a compact subset of ${\mathbb{R}}^{2p}\times \Theta \subset {\mathbb{R}}^{2p+r}$, ${\overrightarrow{a}}^{N\ast }={[a_i^{N\ast }]}_{i=1}^p$, ${\overrightarrow{b}}^{N\ast }={[b_i^{N\ast }]}_{i=1}^p$, which minimizes

$$J^N(\rho )=\sum _{i=1}^m\sum _{j=0}^{n_i}{\mid {\mathcal{Y}}_{i,j}^N({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},\rho )-{\tilde{y}}_{i,j}\mid }^2,$$ (5.5)

where in (5.5), for i = 1, 2, …, m, ${\mathcal{Y}}_{i,j}^N({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},\rho )=\mathcal{C}{(\rho )}^N{x}_{i,j}^N$ is given by (4.7) and (4.9) with $u_j={\tilde{u}}_{i,j}$, j = 0, …, n_i, i = 1, 2, …, m, ${\mathcal{X}}_j$ replaced by ${\mathcal{X}}_{i,j}^N\in {\mathcal{U}}^N$, $\widehat{\mathcal{A}}(\rho )$ replaced by

$${\widehat{\mathcal{A}}}^N(\rho )={\mathcal{T}}^N(\tau ;\rho )=e^{{\mathcal{A}}^N(\rho )\tau }\in \mathcal{L}({\mathcal{U}}^N,{\mathcal{U}}^N),$$

$\widehat{\mathcal{B}}(\rho )$ replaced by ${\widehat{\mathcal{B}}}^N(\rho )={\int }_0^{\tau }{e}^{{\mathcal{A}}^N(\rho )s}{\mathcal{B}}^N(\rho )ds\in \mathcal{L}({\mathbb{R}}^{\mu },{\mathcal{U}}^N)$, $\widehat{\mathcal{C}}(\rho )$ replaced by ${\widehat{\mathcal{C}}}^N(\rho )\in \mathcal{L}({\mathcal{U}}^N,{\mathbb{R}}^{\nu })$, and ${\mathcal{X}}_{i,0}$ replaced by ${\mathcal{X}}_{i,0}^N={\mathcal{J}}^N{\mathcal{X}}_{i,0}\in {\mathcal{U}}^N$. It follows that for i = 1, 2, …, m,

$$x_{i,j+1}^N={\widehat{A}}^N(\rho )x_{i,j}^N+{\widehat{\mathcal{B}}}^N(\rho ){\tilde{u}}_{i,j},{\mathcal{Y}}_{i,j}^N={\mathcal{C}}^N(\rho )x_{i,j}^{N}j=0,1,2,\dots ,$$ (5.6)

with the operators ${\mathcal{A}}^N(\rho )$, ${\mathcal{B}}^N(\rho )$, and ${\mathcal{C}}^N(\rho )$ appearing in (5.6) are as they have been defined above using (5.2)-(5.4).

In the following sections we prove that there exists a subsequence of solutions to the sequence of approximating problems that converges to the solution of our original estimation/optimization problem.

5.2. A Version of the Trotter-Kato Semigroup Approximation Theorem

Our convergence proof is based on a version of the Trotter-Kato semigroup approximation theorem ([5, 21, 26]) that does not require the approximating spaces to be subspaces of the underlying infinite dimensional state space. Banks, Burns and Cliff [1] proved just such a result but unfortunately they do not state their hypotheses in terms of resolvent convergence which is what we require here. Consequently we establish the result in its requisite form here.

Let $\widehat{H}$ be a Hilbert space with norm ∣ · ∣ and let $\{{\widehat{H}}^N\}$ be a sequence of Hilbert spaces, each equipped with norm ∣ · ∣_N. Assume that for each $N\in \mathbb{N}$, ${\widehat{U}}^N$ is a closed (finite dimensional) subspace of $\{{\widehat{H}}^N\}$. Assume that the operators $\widehat{A}$ on $\widehat{H}$, and for each $N\in \mathbb{N}$, ${\widehat{A}}^N$ on ${\widehat{U}}^N$, are in G(M, λ₀) with M and λ₀ independent of N; that is they are the infinitesimal generators of C₀-semigroups $\widehat{S}(t)$ on $\widehat{H}$ and ${\widehat{S}}^N(t)$, on ${\widehat{U}}^N$, respectively, that are uniformly (uniformly in N) exponentially bounded. (We note that if $\widehat{A}$ is obtained from a bounded and coercive sesquilinear form and the ${\widehat{U}}^{N,}\mathrm{s}$ are subspaces with ${\widehat{A}}^N$ defined as the restrictions of $\widehat{A}$ to ${\widehat{U}}^N$, then this latter assumption is easily verified [3, 5].)

Theorem 5.1. Let $\widehat{H}$, ${\widehat{H}}^N$, and ${\widehat{U}}^N$ be Hilbert spaces as defined above. Let ${\mathcal{J}}^N:\widehat{H}\to {\widehat{H}}^N$ be an operator such that $\mathit{Im}({\mathcal{J}}^N)={\widehat{H}}^N$ and ${\mid {\mathcal{J}}^{N}z\mid }_N\le \mid z\mid$. Let ${\mathcal{P}}^N:{\widehat{H}}^N\to {\widehat{U}}^N$ be the canonical projection of ${\widehat{H}}^N$ onto ${\widehat{U}}^N$ and define $P^N≔{\mathcal{P}}^N\circ {\mathcal{I}}^N$. Let $\widehat{A}\in G(M,{\lambda }_0)$ on $\widehat{H}$, and ${\widehat{A}}^N\in G(M,{\lambda }_0)$ on ${\widehat{U}}^N$. Suppose that for some λ ≥ λ₀,

$${\mid P^N{R}_{\lambda }(\widehat{A})z-R_{\lambda }({\widehat{A}}^N)P^{N}z\mid }_N\to 0,asN\to \infty ,$$ (5.7)

for every $z\in \widehat{H}$, where $R_{\lambda }(\widehat{A})={(\lambda I-\widehat{A})}^{-1}$ and $R_{\lambda }({\widehat{A}}^N)={(\lambda I-{\widehat{A}}^N)}^{-1}$ denote respectively the resolvent operators of $\widehat{A}$ and ${\widehat{A}}^N$ at λ. Then

$${\mid P^N\widehat{S}(t)z-{\widehat{S}}^N(t)P^{N}z\mid }_N\to 0,asN\to \infty ,$$ (5.8)

in ${\widehat{H}}^N$, for every $z\in \widehat{H}$ uniformly in t on compact t-intervals.

Proof. For ease of exposition and without loss of generality, let λ₀ = 0. Then, since $\widehat{S}(t)R_{\lambda }(\widehat{A})$ and ${\widehat{S}}^N(t)R_{\lambda }({\widehat{A}}^N)$ are both strongly differentiable in t, we have

$$\frac{d}{dt}\widehat{S}(t)R_{\lambda }(\widehat{A})=\widehat{A}\widehat{S}(t)R_{\lambda }(\widehat{A})=\widehat{S}(t)\widehat{A}{R}_{\lambda }(\widehat{A})=\widehat{S}(t)[\lambda R_{\lambda }(\widehat{A})-I].$$ (5.9)

Then, using an identity for ${\widehat{S}}^N(t)R_{\lambda }({\widehat{A}}^N)$ analogous to (5.9) , we obtain

$$\begin{gathered} \frac{d}{ds}[{\widehat{S}}^N(t-s)R_{\lambda }({\widehat{A}}^N)P^N\widehat{S}(s)R_{\lambda }(\widehat{A})] \\ ={\widehat{S}}^N(t-s)[P^N{R}_{\lambda }(\widehat{A})-R_{\lambda }({\widehat{A}}^N)P^N]\widehat{S}(s). \end{gathered}$$ (5.10)

Then, since

$$\begin{gathered} {\phantom{\mid }{\widehat{S}}^N(t-s)R_{\lambda }({\widehat{A}}^N)P^N\widehat{S}(s)R_{\lambda }(\widehat{A})\mid }_{s=0}^{s=t} \\ =R_{\lambda }({\widehat{A}}^N)[P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]R_{\lambda }(\widehat{A}), \end{gathered}$$ (5.11)

(5.10) and (5.11) imply that

$$\begin{gathered} R_{\lambda }({\widehat{A}}^N)[P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]R_{\lambda }(\widehat{A}) \\ ={\int }_0^t{\widehat{S}}^N(t-s)[P^N{R}_{\lambda }(\widehat{A})-R_{\lambda }({\widehat{A}}^N)P^N]\widehat{S}(s)ds. \end{gathered}$$ (5.12)

Equation (5.12) and $\mid {\widehat{S}}^N(t-s)\mid \le M$ (recall λ₀ = 0), for any $u\in \widehat{H}$, yield

$$\begin{gathered} {\mid R_{\lambda }({\widehat{A}}^N)[P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]R_{\lambda }(\widehat{A})u\mid }_N \\ \le M{\int }_0^t{\mid [P^N{R}_{\lambda }(\widehat{A})-R_{\lambda }({\widehat{A}}^N)P^N]\widehat{S}(s)u\mid }_{N}ds. \end{gathered}$$ (5.13)

By (5.7), we know that the integrand in (5.13) converges to 0 for a fixed s, and also it is bounded by 2M²∣u∣/λ, and therefore, by the Lebesgue Dominated Convergence Theorem, the right-hand side of (5.13) converges to 0 as N → ∞, where the convergence is uniform in t on compact t-intervals.

Letting $v=R_{\lambda }(\widehat{A})u$, and using the fact that $D(\widehat{A})$ is dense in $\widehat{H}$, we have that

$$\mid R_{\lambda }({\widehat{A}}^N)[P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]v{\mid }_N\to 0,asN\to \infty ,$$ (5.14)

for all $v\in \widehat{H}$. Then, since $\mid \widehat{S}(t)\mid \le M$, (5.7) implies that

$$\begin{gathered} \mid R_{\lambda }({\widehat{A}}^N){\widehat{S}}^N(t)P^{N}v-{\widehat{S}}^N(t)P^N{R}_{\lambda }(\widehat{A})v{\mid }_N \\ =\mid {\widehat{S}}^N(t)[R_{\lambda }({\widehat{A}}^N)P^{N}v-P^N{R}_{\lambda }(\widehat{A})v]{\mid }_N\to 0, \end{gathered}$$ (5.15)

and similarly, $\mid {\widehat{S}}^N(t)\mid \le M$ and ((5.7)) imply that

$$\begin{gathered} \mid R_{\lambda }({\widehat{A}}^N)P^N\widehat{S}(t)v-P^N\widehat{S}(t)R_{\lambda }(\widehat{A})v{\mid }_N \\ =\mid [R_{\lambda }({\widehat{A}}^N)P^N-P^N{R}_{\lambda }(\widehat{A})]\widehat{S}(t)v{\mid }_N\to 0. \end{gathered}$$ (5.16)

Combining (5.15), (5.16), and the triangle inequality we get

$$\mid R_{\lambda }({\widehat{A}}^N)[{\widehat{S}}^N(t)P^N-P^N\widehat{S}(t)]v+[P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]R_{\lambda }(\widehat{A})v{\mid }_N\to 0,$$ (5.17)

as N → ∞. Then, because of (5.14), and again by the triangle inequality, we obtain that

$$\mid [P^N\widehat{S}(t)-{\widehat{S}}^N(t)P^N]R_{\lambda }(\widehat{A})v{\mid }_N\to 0,asN\to \infty .$$ (5.18)

Letting $w=R_{\lambda }(\widehat{A})v$, we have $w\in \mathit{Dom}({\widehat{A}}^2)$; and since $\mathit{Dom}({\widehat{A}}^2)$ is dense in $\widehat{H}$, it follows from (5.7), (5.17) and (5.18) that

$$\mid {\widehat{S}}^N(t)P^{N}z-P^N\widehat{S}(t)z{\mid }_N\to 0,asN\to \infty ,$$

for all $z\in \widehat{H}$ uniformly in t on compact t-intervals. □

5.3. Application to the Density Estimation Problem

Let {ρ^N}, ρ ∈ Ξ be such that f^N(q) → f(q), for almost every q ∈ Q, where f^N(q) = f(q; ρ^N) and f(q) = f(q; ρ). Let $\stackrel{‒}{\mathcal{H}}$, $\stackrel{‒}{\mathcal{V}}$, ${\widehat{V}}^N$, ${\mathcal{U}}^N$, ${\mathcal{I}}^N$, $\stackrel{‒}{\mathcal{H}}\to {\widehat{H}}^N$, ${\mathcal{P}}^N:{\mathcal{H}}^N\to {\mathcal{U}}^N$, and ${\mathcal{J}}^N:\stackrel{‒}{\mathcal{H}}\to {\mathcal{U}}^N$ be as they were defined earlier. Set $\mathcal{A}=\mathcal{A}(\rho )$ and consider it to be an operator on $\stackrel{‒}{\mathcal{H}}$ and $\stackrel{‒}{\mathcal{V}}$ by extending f(·, ρ), which is defined on Q, to $\stackrel{‒}{Q}$ by setting it equal to zero on $\stackrel{‒}{Q}\setminus Q$ and let ${\mathcal{A}}^{\mathcal{N}}={\mathcal{A}}^N({\rho }^N)$. Then it follows from Assumptions (i) - (iii) that $\mathcal{A}$ is in G(M, λ₀) on $\stackrel{‒}{\mathcal{H}}$ and ${\mathcal{A}}^{\mathcal{N}}$ is in G(M, λ₀) on ${\mathcal{H}}^N$ with M and λ₀ independent of N.

In the statement of Theorem (5.1), set $\widehat{H}=\stackrel{‒}{\mathcal{H}}$, ${\widehat{H}}^N={\mathcal{H}}^N$, ${\widehat{U}}^N={\mathcal{U}}^N$, $P^N={\mathcal{J}}^N$, $\widehat{A}=\mathcal{A}$, and ${\widehat{A}}^N={\mathcal{A}}^N$. To apply Theorem (5.1) and conclude that in this case, (5.8) holds, we need only verify (5.7). In order to do this, we require the following two additional assumptions

1. There exist positive real numbers γ and δ such that for any ρ ∈ Ξ, we have 0 < γ ≤ f(q; ρ) ≤ δ < ∞ for π(ρ)-a.e. q ∈ Q.

2. For all $w\in \stackrel{‒}{\mathcal{V}}$, there exists $u^N\in {\mathcal{U}}^N$ such that ${\Vert u^N-{\mathcal{J}}^{N}w\Vert }_{{\mathcal{V}}^N}\to 0$ as N → ∞.

We are now able to prove the following theorem.

Theorem 5.2. Let assumptions (i) - (v) be satisfied and let {ρ^N}, ρ ∈ Ξ be such that f^N(q) → f(q), for almost every $q\in \stackrel{‒}{Q}$, where f^N(q) = f(q; ρ^N) and f(q) = f(q; ρ). Then, with the definitions above, the conditions of Theorem (5.1) (and in particular the resolvent convergence specified in (5.7)) are satisfied. Consequently, it follows that

$$\Vert {\mathcal{T}}^N(t;{\rho }^N)P^{N}z-{\mathcal{J}}^N\mathcal{T}(t;\rho )z{\Vert }_{{\mathcal{H}}^N}\to 0,asN\to \infty ,$$ (5.19)

for every $z\in \mathcal{H}$, uniformly in t on compact t-intervals where ${\mathcal{T}}^N=\{{\mathcal{T}}^N(t;{\rho }^N):t\ge 0\}$ is the semigroup on ${\mathcal{H}}^N(\rho )$ given by ${\mathcal{T}}^N(t;{\rho }^N)=e^{{\mathcal{A}}^N({\rho }^N)t}$ and $\mathcal{T}=\{\mathcal{T}(t;\rho ):t\ge 0\}$ is the semigroup on $\mathcal{H}$ and $\stackrel{‒}{\mathcal{H}}$ given by $\mathcal{T}(t;\rho )=e^{\mathcal{A}t}=e^{\mathcal{A}(\rho )t}$.

Proof. First, note that if we can show resolvent convergence for every $z\in \stackrel{‒}{\mathcal{V}}$, then since $\stackrel{‒}{\mathcal{V}}$ is dense in $\stackrel{‒}{\mathcal{H}}$, and ${\mathcal{J}}^N{R}_{{\lambda }_0}(\mathcal{A})$ and $R_{{\lambda }_0}({\mathcal{A}}^N){\mathcal{J}}^N$ are uniformly bounded, the desired resolvent convergence for every $z\in \stackrel{‒}{\mathcal{H}}$ will have been demonstrated. In what follows, for any $\rho =(\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })\in \Xi$, f (·; ρ) is defined on $Q={\prod }_{i=1}^p[a_i,b_i]$, but it can be extended to be defined on $\stackrel{‒}{Q}$ by setting it equal to zero on $\stackrel{‒}{Q}\setminus Q$. We will use this fact frequently below without further remark.

Let $z\in \stackrel{‒}{\mathcal{V}}$ and define $w=R_{{\lambda }_0}(\mathcal{A})z$, and $w^N=R_{{\lambda }_0}({\mathcal{A}}^N){\mathcal{J}}^{N}z$. Suppose also that $u^N\in {\mathcal{U}}^N$ be as in Assumption (v) for $w=R_{{\lambda }_0}(\mathcal{A})z$.

Then, by triangle inequality, we have

$$\begin{gathered} \Vert {\mathcal{J}}^{N}w-w^N{\Vert }_{{\mathcal{V}}^N}\le \Vert {\mathcal{J}}^{N}w-u^N+u^N-w^N{\Vert }_{{\mathcal{V}}^N} \\ \le \Vert {\mathcal{J}}^{N}w-u^N{\Vert }_{{\mathcal{V}}^N}+\Vert u^N-w^N{\Vert }_{{\mathcal{V}}^N}. \end{gathered}$$ (5.20)

Thus, (5.20), Assumption (v) and the continuous embedding of ${\mathcal{V}}^N$ in ${\mathcal{H}}^N$ imply that it is enough to show that ${\Vert u^N-w^N\Vert }_{{\mathcal{V}}^N}\to 0$ as N → ∞. Let z^N = w^N – u^N. Then, since $w^N\in {\mathcal{U}}^N\subset {\mathcal{V}}^N)$,

$$\begin{gathered} \mathcal{A}\left({\rho }^N;w^N,z^N\right)={\langle -{\mathcal{A}}^N{w}^N,z^N\rangle }_{{\mathcal{H}}^N} \\ ={\langle ({\lambda }_{0}I-{\mathcal{A}}^N)R_{{\lambda }_0}({\mathcal{A}}^N){\mathcal{J}}^{N}z,z^N\rangle }_{{\mathcal{H}}^N}-{\lambda }_0{\langle w^N,z^N\rangle }_{{\mathcal{H}}^N} \\ ={\langle {\mathcal{J}}^{N}z,z^N\rangle }_{{\mathcal{H}}^N}-{\lambda }_0{\langle w^N,z^N\rangle }_{{\mathcal{H}}^N}. \end{gathered}$$ (5.21)

Also, since $w\in \mathit{Dom}(\mathcal{A})$,

$$\begin{gathered} \mathcal{a}(\rho ;w,{\mathcal{I}}^{N+}{z}^N)={\langle -\mathcal{A}w,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}} \\ ={\langle ({\lambda }_{0}I-\mathcal{A})R_{{\lambda }_0}(\mathcal{A})z,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}}-{\lambda }_0{\langle w,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}} \\ ={\langle z,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}}-{\lambda }_0{\langle w,{\mathcal{I}}^{N+},z^N\rangle }_{\stackrel{‒}{\mathcal{H}}}, \end{gathered}$$ (5.22)

where ${\mathcal{J}}^{N^{+}}$ denotes the Moore-Penrose generalized inverse [11] of ${\mathcal{J}}^N$. We note that for $\psi \in {\mathcal{H}}^N$, ${\mathcal{J}}^{N^{+}}\psi$ is the function in $\stackrel{‒}{\mathcal{H}}$ that agrees with ψ on Q^N and is zero on $\stackrel{‒}{Q}\setminus Q^N$. Then, from (5.21) and (5.22), we obtain

$$\begin{gathered} \mathcal{a}({\rho }^N;w^N,z^N)-\mathcal{a}(\rho ;w,{\mathcal{I}}^{N+}{z}^N)=\langle -{\mathcal{I}}^{N}z,z^N{\rangle }_{{\mathcal{H}}^N}-{\lambda }_0\langle w^N,z^N{\rangle }_{{\mathcal{H}}^N} \\ -{\langle z,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}}+{\lambda }_0{\langle w,{\mathcal{I}}^{N+}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}}. \end{gathered}$$ (5.23)

Recalling Assumptions (i) and (ii) for the form α(·; ·, ·) on V × V, let ${\tilde{\alpha }}_0$, ${\tilde{\mu }}_0$, ${\tilde{\lambda }}_0$ denote the boundedness and coercivity coefficients for the forms $\mathcal{A}(\cdot ;\cdot ,\cdot )$. Then, using boundedness, coercivity, Assumptions (iv) and (v), Young’s and the Cauchy Schwarz Inequalities, and the continuous embeddings of the space V in the space H (i.e. that there exist a constant k such that ∣ · ∣_H ≤ k∥ · ∥_V) and (5.23), for any ε > 0, we obtain

$$\begin{gathered} {\tilde{\mu }}_0{\Vert z^N\Vert }_{{\mathcal{V}}^N}\le \mathcal{A}({\rho }^N;z^N,z^N)+{\tilde{\lambda }}_0\mid z^N\mid {{}_{\mathcal{H}}}^N \\ =\mathcal{A}({\rho }^N;w^N,z^N)-\mathcal{a}({\rho }^N;u^N,z^N)+{\tilde{\lambda }}_0{\mid z^N\mid }_{{\mathcal{H}}^N}^2 \\ =\mathcal{A}({\rho }^N;w^N,z^N)-\mathcal{a}(\rho ;w,{\mathcal{I}}^{N^{+}}{z}^N) \\ +\mathcal{A}(\rho ;w,{\mathcal{I}}^{N^{+}}{z}^N)-\mathcal{a}({\rho }^N;u^N,z^N)+{\tilde{\lambda }}_0{\mid z^N\mid }_{{\mathcal{H}}^N}^2 \\ ={\langle {\mathcal{I}}^{N}z,z^N\rangle }_{{\mathcal{H}}^N}-{\tilde{\lambda }}_0{\langle w^N,z^N\rangle }_{{\mathcal{H}}^N}-{\langle z,{\mathcal{I}}^{N^{+}}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}}+{\tilde{\lambda }}_0{\langle w,{\mathcal{I}}^{N^{+}}{z}^N\rangle }_{\stackrel{‒}{\mathcal{H}}} \\ +{\int }_{\stackrel{‒}{Q}}(a(q;w,z^N)f(q)-a(q;u^N,z^N)f^N(q))dq+{\tilde{\lambda }}_0{\mid z^N\mid }_{{\mathcal{H}}^N}^2 \\ ={\int }_{\stackrel{‒}{Q}}({\langle z,z^N\rangle }_H(f^N(q)-f(q))dq \\ +{\tilde{\lambda }}_0{\int }_{\stackrel{‒}{Q}}({\langle w,z^N\rangle }_{H}f(q)-{\langle u^N,z^N\rangle }_H{f}^N(q))dq \\ +{\int }_{\stackrel{‒}{Q}}(a(q;w,z^N)f(q)-a(q;u^N,z^N)f^N(q))dq \\ ={\int }_{\stackrel{‒}{Q}}({\langle z,z^N\rangle }_H(f^N(q)-f(q))dq+{\tilde{\lambda }}_0{\int }_{\stackrel{‒}{Q}}{\langle w,z^N\rangle }_H(f(q)-f^N(q))dq \\ +{\tilde{\lambda }}_0{\int }_{\stackrel{‒}{Q}}{\langle w-u^N,z^N\rangle }_H{f}^N(q)dq+{\int }_{\stackrel{‒}{Q}}(a(q;w,z^N)(f(q)-f^N(q))dq \\ +{\int }_{\stackrel{‒}{Q}}a(q;w-u^N,z^N)f^N(q)dq \\ \le {\int }_{\stackrel{‒}{Q}}{\mid z\mid }_H{\mid z^N\mid }_H\mid f^N(q)-f(q)\mid dq+{\tilde{\lambda }}_0{\int }_{\stackrel{‒}{Q}}{\mid w\mid }_H{\mid z^N\mid }_H\mid f(q)-f^N(q)\mid dq \\ +{\tilde{\lambda }}_0{\int }_{\stackrel{‒}{Q}}{\mid w-u^N\mid }_H{\mid z^N\mid }_H{f}^N(q)dq+{\alpha }_0{\int }_{\stackrel{‒}{Q}}{\Vert w\Vert }_V{\Vert z^N\Vert }_V\mid f(q)-f^N(q)\mid dq \\ +{\alpha }_0{\int }_{\stackrel{‒}{Q}}{\Vert w-u^N\Vert }_V{\Vert z^N\Vert }_V{f}^N(q)dq \\ \le \frac{\epsilon k^2}{2\alpha }{\int }_{Q^N}{\Vert z^N\Vert }_V^2{f}^N(q)dq+\frac{1}{2\epsilon }{\int }_{\stackrel{‒}{Q}}{\mid z\mid }_H^2{\mid f^N(q)-f(q)\mid }^{2}dq \\ +\frac{{\tilde{\lambda }}_0\epsilon k^2}{2\alpha }{\int }_{Q^N}{\Vert z^N\Vert }_2^V{f}^N(q)dq+\frac{{\tilde{\lambda }}_0{k}^2}{2\epsilon }{\int }_{\stackrel{‒}{Q}}{\Vert w\Vert }_V^2{\mid f^N(q)-f(q)\mid }^{2}dq \\ +\frac{{\tilde{\lambda }}_0\epsilon k^2}{2}{\int }_{Q^N}{\Vert z^N\Vert }_V^2{f}^N(q)dq+\frac{{\tilde{\lambda }}_0{k}^2}{2\epsilon }{\int }_{Q^N}{\Vert w-u^N\Vert }_V^2{f}^N(q)dq \\ +\frac{{\alpha }_0\epsilon }{2\alpha }{\int }_{Q^N}{\Vert z^N\Vert }_V^2{f}^N(q)dq+\frac{{\alpha }_0}{2\epsilon }{\int }_{\stackrel{‒}{Q}}{\Vert w\Vert }_V^2{\mid f^N(q)-f(q)\mid }^{2}dq \\ +\frac{{\alpha }_0\epsilon }{2}{\int }_{Q^N}{\Vert z^N\Vert }_V^2{f}^N(q)dq+\frac{{\alpha }_0}{2\epsilon }{\int }_{Q^N}{\Vert w-u^N\Vert }_V^2{f}^N(q)dq. \end{gathered}$$ (5.24)

Then, letting $\tilde{c}={\tilde{\mu }}_0-\frac{\epsilon }{2\alpha }\left(k^2({\tilde{\lambda }}_0+1)+(\alpha +1)({\tilde{\lambda }}_0{k}^2+{\alpha }_0)\right)$, it follows from (5.24) that

$$\begin{gathered} \tilde{c}\Vert z^N{\Vert }_{{\mathcal{V}}^N}^2\le \frac{1}{2\epsilon }{\int }_{\stackrel{‒}{Q}}\mid z{\mid }_H^2{f}^N(q)-f(q){\mid }^{2}dq+\frac{{\tilde{\lambda }}_0{k}^2+{\alpha }_0}{2\epsilon }{\int }_{Q^N}\Vert w-u^N{\Vert }_V^2{f}^N(q)dq \\ +\frac{{\tilde{\lambda }}_0{k}^2+{\alpha }_0}{2\epsilon }{\int }_{\stackrel{‒}{Q}}\Vert w{\Vert }_V^2\mid f^N(q)-f(q){\mid }^{2}dq \\ =\frac{1}{2\epsilon }{\int }_{\stackrel{‒}{Q}}\mid z{\mid }_H^2\mid f^N(q)-f(q){\mid }^{2}dq+\frac{{\tilde{\lambda }}_0{k}^2+{\alpha }_0}{2\epsilon }\Vert {\mathcal{J}}^{N}w-u^N{\Vert }_{{\mathcal{V}}^N}^2 \\ +\frac{{\tilde{\lambda }}_0{k}^2+{\alpha }_0}{2\epsilon }{\int }_{\stackrel{‒}{Q}}\Vert w{\Vert }_V^2\mid f^N(q)-f(q){\mid }^{2}dq. \end{gathered}$$ (5.25)

Choosing ε positive, but sufficiently small in (5.25), it follows from Assumption (v) and the hypotheses of the theorem that

$$\Vert w^N-u^N{\Vert }_{{\mathcal{V}}^N}=\Vert z^N{\Vert }_{{\mathcal{V}}^N}\to 0asN\to \infty .$$ (5.26)

Thus (5.26) together with (5.20), and Assumption (v) yield resolvent convergence and the theorem is proved. □

We note that in the proof of Theorem (5.2) we were in fact able to establish resolvent convergence in the ${\mathcal{V}}^N$ norm. Consequently we may conclude that the semigroup convergence in (5.19) is in the ${\mathcal{V}}^N$ norm as well. Moreover, it is not difficult to establish the following corollary to Theorem (5.2).

Corollary 5.1. Under the same hypotheses of Theorem (5.2), we have

$$\begin{gathered} \Vert x_{i,j}^N({\rho }^N)-{\mathcal{J}}^N{x}_{i,j}(\rho ){\Vert }_{{\mathcal{V}}^N}\to 0,asN\to \infty , \\ \mid {\mathcal{Y}}_{i,j}^N({\rho }^N)-{\mathcal{Y}}_{i,j}(\rho ){\mid }_{{\mathbb{R}}^{\nu }}\to 0asN\to \infty , \end{gathered}$$ (5.27)

for every i = 1, 2, …, m, uniformly in j, for j = 0, 1, 2,…, n_i, where ${\mathcal{X}}_{i,j}^N({\rho }^N)$ and ${\mathcal{Y}}_{i,j}^N({\rho }^N)$ are given in (5.6) and ${\mathcal{X}}_{i,j}(\rho )$ and ${\mathcal{Y}}_{i,j}(\rho )$ are given in (4.7).

The assumption that the feasible parameter set Ξ is closed and bounded in ${\mathbb{R}}^{2p+r}$, together with (5.27) in the statement of Corollary (5.1) and Theorem (2.1) then yield the following result.

Theorem 5.3. If, in addition to Assumptions (i)-(v), we assume that the maps ρ ↦ f (q; ρ) from Ξ to $\mathbb{R}$ are continuous for π(ρ) a.e. $q\in \stackrel{‒}{Q}$, then each of the approximating estimation problems admits a solution, ρ^N*. Moreover, the sequence {ρ^N*} has a convergent subsequence, {ρ^N*} with ρ_Nk* → ρ* and ρ* a solution to the original estimation problem.

It is also possible to establish a consistency result for the estimator ${\rho }^{\ast }=({\overrightarrow{a}}^{\ast },{\overrightarrow{b}}^{\ast },{\overrightarrow{\theta }}^{\ast })\in \Xi$. We require the following additional assumptions:

(a) The measurement noise {ε_j,i} is i.i.d. with respect to a probability space {Ω, Σ, P} with $\mathbb{E}[{\epsilon }_{j,i}\Vert P]=0$ and Var[ε_j,i∥P] = σ²,

(b) The feasible set of parameters Ξ is compact (i.e. closed and bounded since it is finite dimensional) and has nonempty interior,

(c) For i = 1, 2, …, n_i = n and nτ = T for some positive integer n and some T > 0, where τ is the sampling time defined in Section 3,

(d) That ${\tilde{y}}_{i,j}={\mathcal{Y}}_{i,j}({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},{\rho }_0)+{\epsilon }_{j,i}$, for some ρ₀ ∈ int{Ξ}, where for i = 1, 2, …, m, ${\mathcal{Y}}_{i,j}({\{{\tilde{u}}_{i,k}\}}_{k=0}^{n_i-1},\rho )$ is given by (4.7) with $u_j={\tilde{u}}_{i,j}$, j = 0, …, n_i, i = 1, 2, …, m, and (4.9), and

(e) For each i = 1, 2, …, m, ρ₀ ∈ Ξ is the unique minimizer of J_i,0 in Xi where

$$J_{i,0}(\rho )={\sigma }^2+{\int }_0^T(\mathcal{Y}(t;{\tilde{u}}_i,{\rho }_0)-\mathcal{Y}(t;{\tilde{u}}_i,\rho ){)}^{2}dt,$$ (5.28)

and $\mathcal{Y}(t;{\tilde{u}}_i,\rho )$ is given by (4.4) -(4.6) with $u={\tilde{u}}_i$.

Then a straight forward application of Theorem 4.2 in [7] can then be used to establish the following lemma and theorem (see [32]).

Lemma 5.1. If in addition to Assumptions (i)-(iv) and (a) (e) above we assume that the maps ρ ↦ f(q; ρ) from Ξ to $\mathbb{R}$ are continuous for π(ρ) a.e. $q\in \stackrel{‒}{Q}$, then there exists an event A ∈ Σ with P(A) = 1 such that for all ω ∈ A and J as given in (5.1) we have

$$\frac{1}{m}\sum _{i=1}^n\{\frac{1}{n}{J}_i(\rho )-J_{i,0}(\rho )\}\to 0,$$

as n, m → ∞ and τ → 0, with nτ = T, uniformly in ρ for ρ ∈ Ξ, where J_i is given by (5.1) and J_i,0 by (5.28).

Theorem 5.4. (Consistency of the estimator ρ*) Let ρ* ∈ Ξ be as defined in (5.1) in Section 5.1. Then under the assumptions of Lemma (5.1) the estimator ${\rho }^{\ast }=({\overrightarrow{a}}^{\ast },{\overrightarrow{b}}^{\ast },{\overrightarrow{\theta }}^{\ast })\in \Xi$ is consistent for ρ₀. That is ρ* → ρ₀ in probability with repsect to the probability measure P, as m, n → ∞, and τ → 0 with ητ = T.

6. Examples and Numerical Results

6.1. The Adjoint Method

The approximating optimization problems are solved numerically by using an iterative gradient-based scheme. Once a basis for the space ^N is chosen, matrix forms of the operators ${\widehat{\mathcal{A}}}^N$, ${\widehat{\mathcal{B}}}^N$, and ${\widehat{\mathcal{C}}}^N$ can be computed. The gradient of J^N(ρ), with respect to the 2p + r parameters in ρ can be computed accurately (in fact exactly with the exception of finite precision arithmetic round-off) and efficiently (which is especially important if the dimension of the approximating system (5.6) and/or the number of parameters is large) using the adjoint method (see [23]). For each i = 1, …, m, set $v_{i,j}^N=2{[{\widehat{\mathcal{C}}}^N]}^T({\widehat{\mathcal{C}}}^N{\mathcal{X}}_{i,j}^N-{\tilde{y}}_{i,j})\in {\mathbb{R}}^{K^N}$, j = 0, …, n_i where K^N is the number of basis elements for ${\mathcal{U}}^N$. Then for each i = 1, …, m, the adjoint systems are defined to be

$$z_{i,j-1}^N=[{\widehat{\mathcal{A}}}^N{]}^T{z}_{i,j}^N+v_{i,j-1}^N,z_{i,n_i}=v_{i,n_i}^N,j=n_i,n_i-1,\dots ,2,1.$$ (6.1)

The gradient of J^N at $\rho =(\overrightarrow{a},\overrightarrow{b},\overrightarrow{\theta })$ can then be computed from

$$\begin{gathered} \overrightarrow{\nabla }{J}^N(\rho )=\sum _{i=1}^m\sum _{j=1}^{n_i}[z_{i,j}^N{]}^T(\frac{\partial {\widehat{\mathcal{A}}}^N}{\partial \rho }{x}_{i,j-1}^N\phantom{)} \\ -({\mathcal{A}}^N{)}^{-1}(\frac{\partial {\mathcal{A}}^N}{\partial \rho }({\mathcal{A}}^N{)}^{-1}({\widehat{\mathcal{A}}}^N-I){\mathcal{B}}^N{\tilde{u}}_{i,j-1}\phantom{)} \\ \phantom{(}\phantom{(}-\frac{\partial {\widehat{\mathcal{A}}}^N}{\partial \rho }{\mathcal{B}}^N{\tilde{u}}_{i,j-1}-({\widehat{\mathcal{A}}}^N-I)\frac{\partial {\mathcal{B}}^N}{\partial \rho }{\tilde{u}}_{i,j-1})) \\ +\sum _{i=1}^m\sum _{j=0}^{n_i}({\mathcal{Y}}_j^N-{\tilde{y}}_{i,j}{)}^T\frac{\partial {\widehat{\mathcal{C}}}^N}{\partial \rho }{x}_{i,j}^N. \end{gathered}$$ (6.2)

Using (6.1) and (6.2) to compute the gradient requires the calculation of the tensor $\frac{\partial {\widehat{\mathcal{A}}}^N}{\partial \rho }$. This can be done using the sensitivity equations. For t ≤ 0 set ${\Phi }^N(t)=e^{{\mathcal{A}}^N(t)}$ from which differentiation yields

$${\stackrel{.}{\Phi }}^N(t)={\mathcal{A}}^N{\Phi }^N(t),{\Phi }^N(0)=I.$$ (6.3)

Then, setting ${\Psi }^N(t)=\frac{\partial {\Phi }^N(t)}{\partial \rho }$, differentiating (6.3) with respect to ρ, and interchanging the order of differentiation, we obtain

$${\stackrel{.}{\Psi }}^N(t)={\mathcal{A}}^N{\Psi }^N(t)+\frac{\partial {\mathcal{A}}^N}{\partial \rho }{\Phi }^N(t),{\Psi }^N(0)=0.$$ (6.4)

Combining (6.3) and (6.4), and solving the resulting system, we obtain

$$\left[\begin{array}{c}\hfill {\Psi }^N(t)\hfill \\ \hfill {\Phi }^N(t)\hfill \end{array}\right]=exp\left(\left[\begin{array}{cc}\hfill {\mathcal{A}}^N\hfill & \hfill \partial {\mathcal{A}}^N∕\partial \rho \hfill \\ \hfill 0\hfill & \hfill {\mathcal{A}}^N\hfill \end{array}\right]\tau \right)\left[\begin{array}{c}\hfill 0\hfill \\ \hfill I\hfill \end{array}\right]$$ (6.5)

Setting t = τ in (6.5), we obtain that $\frac{\partial {\widehat{\mathcal{A}}}^N}{\partial \rho }={\Psi }^N(\tau )$.

To illustrate our approach, we consider the case of a one dimensional heat/diffusion equation on the interval [0, 1] with random (thermal) diffusivity and two different sets of boundary conditions. Consider the partial differential equation, boundary conditions and output operator given by

$$\frac{\partial x}{\partial t}(t,\eta )=q_1\frac{{\partial }^{2}x}{\partial {\eta }^2}(t,\eta ),0<\eta <1,t>0,$$ (6.6)

$${\mathrm{\Gamma }}_{D}x(t,\cdot )=x(t,0)=0,t>0,$$ (6.7)

$${\mathrm{\Gamma }}_{R}x(t,\cdot )=q_1\frac{\partial x}{\partial \eta }(t,0)-x(t,0)=0,t>0,$$ (6.8)

$${\mathrm{\Gamma }}_{1}x(t,\cdot )=\frac{q_1}{q_2}\frac{\partial x}{\partial \eta }(t,1)=u(t)t>0,$$ (6.9)

$$x(0,\eta )=0,0<\eta <1,$$ (6.10)

$$y(t)=x(t,{\eta }_0),t>0,$$ (6.11)

where 0 < η₀ < 1. In the examples below, we consider the parameterized family of probability density functions defined as follows.

Definition 6.1. Let φ(q; θ), $q\in {\mathbb{R}}^n$ be a member in an exponential family [12], and let Φ denote its cumulative distribution function. Let θ represent a vector of parameters, and let $D\subset {\mathbb{R}}^n$ be a bounded region to which φ will be restricted. Then define Φ_D(θ) = ∫_D φ(q; θ)dq. Then the family of pdfs, f (·, ρ) given by

$$f(q;\rho )=\frac{\phi (q;\theta ){\chi }_D(q)}{{\Phi }_D(\theta )}=\frac{1}{{\Phi }_D(\theta )}h(q)c(\theta )exp\left(\sum _{i=1}^k{w}_i(\theta )t_i(q)\right){\chi }_D(q)$$

where the parameters ρ include the parameters θ and parameters $\overrightarrow{a}$ and $\overrightarrow{b}$ to describe the domain D, is called a truncated exponential family.

It is clear that this family of densities satisfies Assumption (iv) and the hypotheses of Theorem (5.1).

All of the numerical results presented here use simulation data. Our studies involving actual experimental/clinical data are discussed elsewhere (see [32]). The simulated data was generated by first sampling the target distribution to obtain 100 samples q of $\mathcal{q}$. A spline based Galerkin approximation to the system (6.6) -(6.11) using a 128 equally spaced point grid on [0,1] was then solved using each $\mathcal{q}$-sample. The resulting 100 output signals were then averaged at each time point. The approximating estimation problems were all solved on either MAC or PC laptops using the Matlab optimization toolbox routine FMINCON for constrained optimization. Gradients were computed using either FMINCON built-in finite differencing or the adjoint method, (6.1)-(6.5). Which method was used had only a negligible effect on the results. The input signal used was u(t) = ∣cos(t)∣χ_[0,2](t), t ∈ [0, 20], and the sampling interval was τ = 0.1. In all of our examples below, the admissible parameter space Q is assumed to be either in ${\mathbb{R}}^{+}$ in the case of the uni-variate examples, or in the fist quadrant of the plane ${\mathbb{R}}^2$ in the bivariate examples. Consequently when the approximating optimization problems were solved, the lower bounds for the supports of the random parameters, a and c, were constrained to be strictly positive. This is based on the requirements of the physical model (6.6)-(6.11) and the assumption that properties (i)-(iii) in Section 3 hold.

6.2. Examples 6.1,6.2 and 6.3; One Random Parameter; Truncated Uniform, Exponential and Normal Distributions

In this series of examples we consider the system (6.6),(6.7),(6.9)-(6.11) with q₁ random and q₂ = 1. In this case we have q = q₁ ∈ Q = [a, b], W = [φ ∈ H²(0, 1), Γ_Dφ = 0}, $H=H_L^1(0,1)=\{\phi \in H^1(0,1),{\mathrm{\Gamma }}_D\phi =0\}$, Dom(A(q)) = [φ ∈ V : Γ_1φ = 0}, and Γ(q) = Γ₁. It follows that

$$a(q;\phi ,\psi )=q{\int }_0^1{\phi }^{\prime }(\eta ){\psi }^{\prime }(\eta )d\eta ,\phi ,\psi \in V,$$

and ⟨b(q), ψ⟩_V*,V = ⟨b, ψ⟩_V*,V = ψ(1) = δ(· − 1), ψ ∈ V, and ⟨c(q), ψ⟩_V*,V = ⟨c, ψ⟩_V*,V = ψ(1/3), ψ ∈ V, where in this case η₀ = 1/3. Standard arguments [3, 5] show that Assumptions (i)-(iii) are satisfied.

To carry out the finite dimensional discretization, we let n, m be positive integers and set N = (n, m). In this case we have either D = [a, b] (uniform and normal) or D = [0, R] (exponential). In what follows we describe the q or Q discretization for the uniform and normal cases; the exponential is similar. The basis for the approximating subspaces ${\mathcal{U}}^N$ were taken to be tensor products of the standard linear spline basis elements ${\phi }_i^n$ corresponding to the uniform mesh $\{0,\frac{1}{n},\frac{2}{n},\dots ,\frac{n-1}{n},1\}$ on [0, 1], and the characteristic function basis ${\chi }_j^m$ for the interval [a, b]. The j^th element corresponds to the j^th sub-interval $[a+(j-1)\frac{b-a}{m}),a+j\frac{b-a}{m})$, j = 1, 2, …, m. In this way ${\mathcal{U}}^N=\mathit{span}\{{\xi }_{i,j}^N\}$, i = 1, 2, …, n, ^j = 1, 2, …, m where ${\xi }_{i,j}^N(\eta ,q)={\phi }_i^n(\eta ){\chi }_j^m(q)$, η ∈ [0, 1], q ∈ [a, b] with $\mathit{dim}({\mathcal{U}}^N)=nm$. Using standard estimates [29] it is not difficult to show that Assumption (v) holds.

Re-numbering ${\xi }_{i,j}^{N,}\mathrm{s}$ so that ${\xi }_{i,j}^N={\xi }_k^N$ where k = (i − 1)n + j and letting ${\Psi }_k^N={[{\psi }_i^N]}_{i=1}^{nm}\in {\mathbb{R}}^{nm}$, the matrix representation for the operators ${\mathcal{A}}^N$ are given by $[{\mathcal{A}}^N]=-{(M^N)}^{-1}{K}^N$ with

$$\begin{gathered} M_{r,s}^N=M_{r,s}^N(a,b,\theta )=\langle {\xi }_r^N,{\xi }_s^N{\rangle }_{\mathcal{H}} \\ ={\int }_a^b{\int }_0^1{\xi }_r^N{\xi }_s^{N}f(q;a,b,\theta )d\eta dq={\int }_a^b{\chi }_j^m{\chi }_l^{m}f(q;a,b,\theta )dq{\int }_0^1{\phi }_i^n{\phi }_k^{n}d\eta , \end{gathered}$$

$$\begin{gathered} K_{r,s}^N=K_{r,s}^N(a,b,\theta )=\mathcal{A}(q;{\xi }_r^N,{\xi }_s^N)={\int }_a^{b}q{\int }_0^1\frac{\partial {\xi }_r^N}{\partial \eta }\frac{\partial {\eta }_s^N}{\partial \eta }f(q;a,b,\theta )d\eta dq \\ ={\int }_a^{b}q{\chi }_j^m{\chi }_l^{m}f(q;a,b,\theta )dq{\int }_0^1{\phi }_i^{n\prime }{\phi }_k^{n\prime }d\eta , \end{gathered}$$

where r = (j − 1)n + i, s = (l − 1)n + k, i, k = 1, 2, …, n, j, l = 1, 2, …, m.

We also have

$$\begin{gathered} B_r^N=B_r^N(a,b,\theta )={\int }_a^b{\xi }_r^N(1,q)f(q;a,b,\theta )dq={\phi }_i^n(1){\int }_a^b{\chi }_j^{m}f(q;a,b,\theta )dq, \\ {C}_s^N(a,b,\theta )={\int }_a^b{\xi }_s^N(1∕3,q)f(q;a,b,\theta )dq-{\phi }_k^n(1∕3){\int }_a^b{\chi }_l^m(q)f(q;a,b,\theta )dq, \end{gathered}$$

r, s = 1, 2, …, nm, r = (j − 1)n + i, s = (l − 1)n + k, i, k = 1, 2, …, n, j, l = 1, 2, …, m.

With the density f = f₀(·; ρ) = f₀(·; (a, b, θ)) as given in Definition (6.1) above, if we define

$$f_1(\alpha ,\beta ;\rho )={\int }_{\alpha }^{\beta }f(q;\rho )dq\text{and}{f}_2(\alpha ,\beta ;\rho )={\int }_{\alpha }^{\beta }qf(q;\rho )dq,$$

it is a straightforward, albeit somewhat tedious, exercise to compute the partial derivatives $\frac{\partial f_i}{\partial \alpha }$, $\frac{\partial f_i}{\partial \beta }$, $\frac{\partial f_i}{\partial \theta }$, $\frac{\partial f_i}{\partial a}$, $\frac{\partial f_i}{\partial b}$, i = 0, 1, 2. These partial derivatives show up in the matrices that appear in the adjoint equations (6.1)-(6.5). We tested our scheme on truncated uniform (ρ = (a, b)), exponential (ρ = (R, θ)) and normal (ρ = (a, b, μ, σ)) distributions. Our results are shown in Table (6.1) and Figure (6.1) below. In panels (a) - (c) of Figure (6.1), we have plotted the converged estimated population models together with the data and the 75% credible band for the truncated uniform, exponential and normal densities. The credible bands can be obtained directly from the solution to the population model. Indeed, $\mathcal{q}$ is sampled using the estimated distribution and then $C(\mathcal{q}){\mathcal{X}}_j^N(\cdot ,\mathcal{q})$ is evaluated at the sample q’s where ${\mathcal{X}}_j^N$ is given by (5.6). Now the q dependence of the solution to the population model is only valid π almost everywhere and our convergence framework is an L₂ (in q) theory. Consequently, pointwise evaluation is, strictly speaking, undefined. However, the results appear to be useful so we have included them. We are currently working on an extension of the results presented here that involves introducing parabolic regularization in q. This will potentially allow us to justify pointwise evaluation in q of the population model to obtain credible band. It is interesting to note that the credible band for the exponential distribution is quite wide, almost to the point of making the population model not that useful. This is because the exponential distribution, especially one with a mean and variance of μ = 1/θ = 3, has a rather “fat” tail. Panels (d) and (f) of Figure (6.1) show the converging estimated pdfs for the truncated exponential and normal distributions, respectively. Panel (e) shows how the output of the population model compares to the data when the resolution of the finite element discretizations of q and η and the truncation point of the densities are varied. It appears from the figure that it is the q discretization that determines the rate of convergence, while a rather coarse η discretization seems to suffice. We believe that this explains the slow convergence of θ (the exponential parameter) and σ (the standard deviation of the normal) observed in Table (6.1) and panel (f) of Figure (6.1). The truncation of the density appears to have only a negligible effect. We are currently investigating whether using smoother first order splines for the q elements produces improved estimates and more rapid convergence.

Table 6.1:

Convergence results for Examples 6.1, 6.2 and 6.3; estimation of the parameters in truncated uniform, exponential and normal distributions.

N			Uniform		Exponential		Normal
n	m		a*	b*	θ*	R*	a*	b*	μ*	σ*
4	4		1.76	4.27	2e-5	3.61	2.61	5.44	4.05	0.62
8	8		1.91	4.05	4e-5	3.81	2.29	5.42	4.01	0.40
16	16		1.94	4.00	0.20	4.34	2.17	5.42	4.01	0.37
32	32		1.95	3.99	0.30	5.95	2.15	5.42	4.00	0.35
64	64		1.96	3.99	0.30	11.08	2.14	5.42	4.00	0.35
True Values			2	4	1/3	—	—	—	4	0.25

Figure 6.1:

Top row, starting from the left: Data, converged estimated population model and 75% credible band for (a) Example 6.1 Truncated uniform distribution; (b) Example 6.2 Truncated exponential distribution; (c) Example 6.3 Truncated normal distribution. Bottom row, starting from the left: (d) Example 6.2 Converged pdfs for truncated exponential distribution; (e) Example 6.2 Data and Estimated population model for various values of R, n and m; (f) Example 6.3 Converged pdfs for truncated normal distribution.

6.3. Example 6.4; Two Random Parameters; Truncated Bi-variate Normal Distribution

In this example we consider the system (6.5)-(6.11), but instead of the Dirichlet boundary condition (6.2) at η = 0, we take the Robin boundary condition (6.3) at η = 0. In this case, q = [q₁, q₂] is the vector of random parameters with q ∈ D = Q = [a, b] × [c, d], H = L²(0, 1), V = H¹(0, 1), W = H²(0, 1), and Dom(A(q)) = {φ ∈ H²(0, 1) : Γ_Rφ = 0, Γ_1φ = 0} and Γ(q) = Γ₁. The sesquilinear form on V × V is given by $a(q;\phi ,\psi )=q_1{\int }_0^1{\phi }^{\prime }{\psi }^{\prime }d\eta +\phi (0)\psi (0)$ with < b(q), ψ >_V*,V = q₂ψ(1) = q₂ψ(· − 1), ψ ∈ V, and < c(q), ψ >_V*,V=< c,ψ >_V*,V = ψ(0), ψ ∈ V where we have set η₀ = 0. In this case N = (n, m₁, m₂), where n is again the level of discretization of the space variable η and m_i is the level of discretization of q_i, i = 1, 2. Once again the approximating subspaces were constructed using tensor products, ${\mathcal{U}}^N=\mathit{span}\{{\xi }_{i,j,k}^N\}$, i = 0, 1, 2, …, n, j = 1, 2, …, m₁, k = 1, 2, …, m₂ where ${\xi }_{i,j,k}^N(\eta ,q_1,q_2)={\phi }_i^n(\eta ){\chi }_j^{m_1}(q_1){\chi }_k^{m_2}(q_2)$, η ∈ [0, 1], q₁ ∈ [a, b], q₂ ∈ [c, d] with $\mathit{dim}({\mathcal{U}}^N)=(n+1)m_1{m}_2$.

In this example the truncated exponential family was based on the bivariate normal. Once again, it is possible to compute all the partial derivatives (although of course their evaluation requires the numerical evaluation of single and double integrals) that are required to form the matrices that appear in the state and adjoint equations (6.1)-(6.5). We obtained simulated data by generating samples for $\mathcal{q}$ from a $N(\stackrel{‒}{\mu },\stackrel{‒}{\Sigma })$ distribution with $\stackrel{‒}{\mu }=\left[\begin{array}{c}\hfill 12\hfill \\ \hfill 10\hfill \end{array}\right]$ and $\stackrel{‒}{\Sigma }=\left[\begin{array}{cc}\hfill 9\hfill & \hfill 3\hfill \\ \hfill 3\hfill & \hfill 5\hfill \end{array}\right]$.

Our results are shown in Table (6.2) and Figure (6.2), where it can be seen that we obtained reasonably good approximations to the actual parameters that we used to simulate the data. We parameterized the covariance matrix as Σ = L^TL, where the 2 × 2 matrix L is upper triangular with L₁₁ and L₂₂ both positive so as to guarantee that at each step in the optimization, Σ is positive definite symmetric. The plot of the optimal joint density in the left hand panel of Figure (6.2) correspond to n = 16 and m₁ = m₂ = 8. In the right hand panel of Figure (6.2) we have plotted the output of the fit population model and the 75% credible band. Once again, we believe that the rate of convergence could be improved by using linear splines rather than piece-wise constant elements to discretize the random parameters q.

Table 6.2:

Convergence results for Example 6.4; estimation of the parameters in truncated bivariate normal distribution.

n	m1	m2		a*	b*	c*	d*	μ*	σ*
4	8	8		5.88	18.15	4.85	14.63	$\left[\begin{array}{c}\hfill 11.72\hfill \\ \hfill 9.88\hfill \end{array}\right]$	$\left[\begin{array}{cc}\hfill 12.13\hfill & \hfill 5.76\hfill \\ \hfill 5.76\hfill & \hfill 7.35\hfill \end{array}\right]$
8	8	8		5.67	18.35	5.17	14.46	$\left[\begin{array}{c}\hfill 11.68\hfill \\ \hfill 9.87\hfill \end{array}\right]$	$\left[\begin{array}{cc}\hfill 10.15\hfill & \hfill 4.04\hfill \\ \hfill 4.04\hfill & \hfill 5.97\hfill \end{array}\right]$
16	8	8		5.79	18.17	5.06	14.66	$\left[\begin{array}{c}\hfill 11.67\hfill \\ \hfill 9.86\hfill \end{array}\right]$	$\left[\begin{array}{cc}\hfill 9.29\hfill & \hfill 3.03\hfill \\ \hfill 3.03\hfill & \hfill 5.21\hfill \end{array}\right]$

Figure 6.2:

Left hand panel: Example 6.4 Estimated bivariate normal joint density with n = 16 and m₁ = m₂ = 8; Right hand panel: Example 6.4 Data, estimated population model and 75% credible band for truncated bivariate normal distribution.

7. Concluding Remarks

We are currently working on a number of applications and extensions of the results presented here. Specifically, we are looking at applying our approach to actual experimental and clinical BrAC and TAC data collected in both the lab/clinic and the field using two different transdermal alcohol biosensors from a number of different individuals that include several drinking episodes occurring over a time period of several days. We are developing deconvolution schemes based on population models fit using the approach discussed here that, given an output signal, will provide a population based estimate for the input together with credible bands obtained directly from the deconvolved input signal and not requiring simulation. We are also looking at extensions of the ideas presented here to the solution of the LQR and LQG compensator problems wherein the infinite dimensional linear regularly dissipative dynamics and quadratic performance index involve random parameters.

In our treatment here, we assumed that the probability measures describing the distribution of the random parameters were defined in terms of parameterized families of joint density functions. We are looking at developing numerical schemes and an associated convergence theory for estimating the shape of the density directly. We also hope to be able to apply the convergence theory based on the Prohorov metric on a space of measures developed in [7] more directly to the class of problems that we have discussed here. More precisely, we would like to be able to eliminate the assumption that the measures are defined in terms of a density, and estimate the measure directly. We believe that such a theory may be possible by assuming that our approximating subspaces are required to satisfy additional regularity (i.e. smoothness) assumptions; in particular that they are required to be contained in the domain of the operator. Then by making use of a slightly different version of the Trotter-Kato semigroup approximation theorem (see, for example, [1]) we believe it may now be possible to verify the hypotheses of the more general convergence theorem established in [7] for the estimation of the probability measures directly, rather than by estimating an associated density.