A Semi-Infinite Programming based algorithm for determining T-optimum designs for model discrimination

Abstract

T-optimum designs for model discrimination are notoriously difficult to find because of the computational difficulty involved in solving an optimization problem that involves two layers of optimization. Only a handful of analytical T-optimal designs are available for the simplest problems; the rest in the literature are found using specialized numerical procedures for a specific problem. We propose a potentially more systematic and general way for finding T-optimal designs using a Semi-Infinite Programming (SIP) approach. The strategy requires that we first reformulate the original minimax or maximin optimization problem into an equivalent semi-infinite program and solve it using an exchange-based method where lower and upper bounds produced by solving the outer and the inner programs, are iterated to convergence. A global Nonlinear Programming (NLP) solver is used to handle the subproblems, thus finding the optimal design and the least favorable parametric configuration that minimizes the residual sum of squares from the alternative or test models. We also use a nonlinear program to check the global optimality of the SIP-generated design and automate the construction of globally optimal designs. The algorithm is successfully used to produce results that coincide with several T-optimal designs reported in the literature for various types of model discrimination problems with normally distributed errors. However, our method is more general, merely requiring that the parameters of the model be estimated by a numerical optimization.

1. Introduction

Professor Jack Kiefer was an early proponent of using a rigorous mathematical framework to find optimal experimental designs for solving practical problems. In Kiefer [24,25] he introduced continuous designs, in which the design is represented by a measure. As a result, the problems of the dependence of the structure of the design on sample size are avoided. He advocated using such designs for practical reasons; research in this area has continued, largely motivated by rising experimental costs and the need to use resources more efficiently. Book length treatments of this topic include Pukelsheim [34], Fedorov and Hackl [16], Uciński [41], Atkinson et al. [4], Berger and Wong [7] and Fedorov and Leonov [17]. Early applications of optimal designs were concentrated in the engineering, manufacturing and industrial sectors but applications are increasingly also seen in the biomedical and social sciences.

Optimal designs can depend sensitively on the assumed model. They can lose substantial efficiency if the assumed model is wrong. In practice, the underlying model is unknown and frequently a few plausible alternative models are considered for studying the problem at hand. An optimal discrimination design provides the best strategy for collecting observations to identify the true model among those postulated. Optimal design problems for estimating model parameters are quite well studied but the search for the optimal discrimination design has received considerably less attention. One reason is that finding an optimal discrimination design is an appreciably more difficult task than finding a D-optimal design for estimating model parameters [45]. Unlike D-optimality, we now have an optimality criterion that requires two levels of optimization. To date, effective algorithms for finding these optimal designs for a general regression model remain elusive.

The theoretical framework for experimental design for model discrimination was established in a series of papers, such as Fedorov and Malyutov [18], Atkinson and Cox [3], Atkinson and Fedorov [5,6]. The criterion used for model discrimination is commonly known as T-optimality. The typical setup assumes that we want to discriminate between two parametric models, one of which is a fully parameterized “true model” and the other a “test model” with unknown parameters. The T-optimal design maximizes the lack of fit sum of squares for the second model by maximizing the minimal lack of fit sum of squares arising from a set of plausible values of the unknown parameters. Additional theoretical developments can be found in Ponce de Leon and Atkinson [33], Dette [9], Fedorov and Hackl [16], Wiens [46] and Dette and Titoff [12]. López-Fidalgo et al. [29] extend the method to models in which the errors of observation do not follow a normal distribution. T-optimality has been applied to discriminate among various classes of models, ranging from polynomial models [5,11], to Fourier regression models [10], Michaelis–Menten kinetic models [30], enzyme kinetics [2] and dynamic systems described by sets of ordinary differential equations [41,27,39].

There are analytical descriptions of T-optimal designs for only the simplest situations because of the complexity of the optimization problem. The algorithms commonly used to find T-optimal designs are based on modifications of the Wynn–Fedorov algorithm, which were initially proposed for D-optimal designs; see for example, Atkinson and Fedorov [5]. The method requires a user-selected starting design to initiate the search process before it iterates by sequentially adding one or more selected new points from the design space to the current design. At each iteration a new design is formed by mixing the new point or points appropriately chosen with the current design. The generated design accumulates many points or clusters of points over time and a judicious collapsing of these points into a smaller number of distinct points is periodically required. These are the core steps in the Wynn–Fedorov algorithm formed by aggregating ideas of Wynn [47] and Fedorov [15] and commonly used in computer algorithms for finding different types of optimal designs such as D_s-optimal designs for estimating a selected subset of the model parameters or L-optimal designs for estimating a selected linear function of the model parameters.

Two other approaches have been employed for determining optimal discrimination designs. Dette and Titoff [12] suggest the Remes algorithm from numerical approximation theory and demonstrated the method for problems with a single explanatory variable. Atkinson [2] employs a Quasi-Newton algorithm for convex optimization after applying a transformation on the design region and design weights to ensure that all constraints are satisfied. See also Atkinson et al. [4, Section 9.5] where more details on the method and examples can be found. However, both methods seem somewhat specialized and may not extend to find optimal discrimination designs for more general problems.

Algorithms based on Semi-Infinite Programming (SIP), a branch of mathematical programming, are becoming increasingly popular for solving the minimax programs in computer science, engineering and economics [37]. Several algorithms belonging to exchange methods, discretization methods and local reduction methods have been developed [36]. Coupled with global nonlinear programming (NLP) solvers, they are able to solve minimax programs of moderate dimension. Interestingly, there are only a couple of applications of mathematical programming SIP-based approaches to find minimax-type optimal designs even though the approach provides a general framework and a systematic approach that is guaranteed find such optimal designs. Our goal in this paper is to apply SIP-based algorithms to systematically find optimal discrimination designs and demonstrate their effectiveness using several examples for a variety of situations. Only non-sequential experimentation is considered here; readers interested in a sequential approach to design a study for model discrimination can refer to Atkinson and Fedorov [5,6].

Gribik and Kortanek [21] established a theoretical and general framework for searching minimax designs via SIP. žakovíc and Rustem [48] found minimax D-optimal designs and Duarte and Wong [14] found various types of minimax optimal designs using SIP based on an exchange method. Kuczewski [27] and Skanda and Lebiedz [39] used a SIP algorithm to find T-optimal designs for dynamic models using algorithms similar to that proposed by žakovíc and Rustem [48] for general minimax problems.

Uciński and Bogacka [42] used a SIP based algorithm to find T-optimal designs for dynamic models. The SIP procedure relies on the relaxation paradigm proposed by Shimizu and Aiyoshi [38] for minimax problems. All optimization problems included in the SIP procedure are solved with a global solver employing a stochastic NLP solver with an adaptive random search scheme to generate initial solutions. There seems to be no application of SIP to finding T-optimal designs for discriminating between algebraically specified models. Uciński and Bogacka [42], Kuczewski [27] and Skanda and Lebiedz [39] deal with dynamic models and aim to determine the optimal discrimination design in the time domain (time instants where samples are to be gathered). Our paper aims to present and test a SIP based algorithm for finding T-optimal designs for algebraic models, both linear and nonlinear. It shares several properties with the procedure proposed by Uciński and Bogacka [42]. We include a check from the equivalence theorem which allows us to automate the finding of the optimal number of support points.

Section 2 provides the background, and introduces the T-optimality criterion along with a practical tool for checking whether a design is optimal among all designs on the given design space. It also presents the conceptualization of the minimax program representing the T-optimality criterion as a SIP, and briefly reviews the exchange method for handling semi-infinite programs. Section 3 applies the SIP based algorithm to find T-optimal designs and an automated procedure for confirming the optimality of the SIP-generated design. We report these T-optimal designs for various discrimination problems in Section 4 and offer a conclusion in Section 5.

2. Background

This section is divided into two parts. The first discusses the statistical setup and use of continuous designs as a practical tool to solve general design problems. The second part provides background on SIP-based methods and how they relate to finding an optimal discrimination design and more generally solving minimax design problems.

2.1. Continuous designs

In this paper, we focus on continuous designs on a given compact design space of the regressors 𝒳 ⊂ ℝ^n_x. A continuous design is characterized by the number of design points it has from the design space 𝒳, the locations of the points and the proportions of the total number of observations n to be taken at each of the design points. Let x_i ∈ 𝒳 be the ith design point or support point of the design, let k be the number of design points and let w_i be the proportion of observations to be taken at x_i, i = 1, …, k. Clearly, w_i is positive and less than unity (unless k = 1) and w₁ + w₂ +···+ w_k = 1. The total sample size n is usually predetermined by cost considerations. Continuous designs have continuous weights in w_i ∈ [0, 1] which lead naturally to the formulation of the optimal design problem as a mathematical program with convex properties. Advantages of working with continuous designs are that they are easier to find and understand than exact designs that depend on n. We denote such a continuous design with k points by

$$\xi =\left(\begin{array}{ccccc}{x}_1& \cdots & x_i& \cdots & x_k\\ w_1& \cdots & w_i& \cdots & w_k\end{array}\right)$$

and denote the set of all continuous designs with k points on 𝒳 by Ξ ≡ 𝒳^k × [0, 1]^k.

For exact designs, we require that all n × w_i’s are positive integers. In this case, we would have to solve a much harder non-convex optimization problem. Pukelsheim and Rieder [35] describe an efficient method for rounding a continuous design to obtain a nearly optimum exact design of size n. Goos and Jones [20] give examples of finding exact D-optimal designs using a coordinate-exchange algorithm.

For model discrimination design problems, we seek a continuous design that is efficient for identifying the best fitting model from a given class of models. When there are two models and the outcome variable is Y, we designate one as the “true model” η_t (x, θ₁) = E(Y |x, θ₁( and the other as the “test model” η₂(x, θ₂( = E(Y |x, θ₂(. The vectors of model parameters θ₁ and θ₂ may have different dimensions, but lie in known sets Θ₁ and Θ₂, i.e. θ₁ ∈ Θ₁ ⊂ ℝ^p₁ and θ₂ ∈ Θ₂ ⊂ ℝ^p₂. Following convention, we assume the “true model” is fully parameterized and so the dependence on θ₁ can be discarded and we may write its mean function simply as η_t (x(.

A common design criterion called T-optimality for model discrimination was proposed by Atkinson and Fedorov [5] and Atkinson et al. [4]. The T-optimal design is defined by:

$$\begin{array}{l}{\xi }_T=arg\underset{\xi \in \mathit{\Xi }}{max}\underset{{\theta }_2\in {\mathit{\Theta }}_2}{min}{\int }_{\mathcal{X}}{[{\eta }_t(x)-{\eta }_2(x,{\theta }_2)]}^2\xi (\mathrm{d}x)\\ =arg\underset{\xi \in \mathit{\Xi }}{min}\underset{{\theta }_2\in {\mathit{\Theta }}_2}{max}-{\int }_{\mathcal{X}}{[{\eta }_t(x)-{\eta }_2(x,{\theta }_2)]}^2\xi (\mathrm{d}x).\end{array}$$ (1)

Employing results from Rustem and Howe [37], problem (1) is equivalent to the bilevel program

$$\begin{array}{l}{\xi }_T=arg\underset{\xi \in \mathit{\Xi }}{min}-{\int }_{\mathcal{X}}{[{\eta }_t(x)-{\eta }_2(x,{\theta }_2^{\ast })]}^2\xi (\mathrm{d}x)\\ \text{s.t.}\sum _{i=1}^k{w}_i=1\\ {\theta }_2^{\ast }=arg\underset{{\theta }_2\in {\mathit{\Theta }}_2}{max}-{\int }_{\mathcal{X}}{[{\eta }_t(x)-{\eta }_2(x,{\theta }_2)]}^2\xi (\mathrm{d}x),\end{array}$$ (2)

showing that the T-optimality criterion can be equivalently viewed as a maximin, a minimax or a bilevel optimization problem with the outer program having convex properties and the inner problem being concave or convex. An important quantity in the above definition is the least favorable parametric configuration ${\theta }_2^{\ast }$ in Θ2, which is frequently problematic to determine numerically and presents a constant source of difficulty for finding the optimal discrimination design, and more generally for minimax or maximin optimal designs in practice.

The search for the optimal discrimination design ξ_T is nested within the number of support points of the design. To avoid the complexity of simultaneously find the design and the number of support points, a nonconvex optimization problem, we fix k and start the search over all k-point designs. The resulting design ${\xi }_T^k$ may or may not be optimal among all designs on Ξ. An equivalence theorem similar to those given in Kiefer and Wolfowitz [26] and Kiefer [25] is then used to check whether ${\xi }_T={\xi }_T^k$. The mathematical program to solve the problem is:

$$\begin{array}{l}\mathit{\Delta }({\xi }_T^k)=\underset{\xi \in \mathit{\Xi }}{min}\underset{{\theta }_2\in {\mathit{\Theta }}_2}{max}-\sum _{i=1}^k{[{\eta }_t(x_i)-{\eta }_2(x_i,{\theta }_2)]}^2{w}_i\\ \text{s.t.}\sum _{i=1}^k{w}_i=1.\end{array}$$ (3)

A common choice for initializing k is the number of parameters in the model plus one. A theoretical justification for the choice of the value of k is possible only in specialized settings. For example, Dette and Titoff [12] proved that, for nested polynomials in one variable, k = p₂ + 1. Our numerical results in Section 4 support such a value for k. For T-optimality, the theorem asserts that the design ${\xi }_T^k$ is optimal among all designs on 𝒳 if and only if

$${[{\eta }_t(x)-{\eta }_2(x,{\theta }_2^k)]}^2\le -\mathit{\Delta }({\xi }_T^k),\forall x\in \mathcal{X},$$ (4)

with equality at the support points of ${\xi }_T^k$ and ${\theta }_2^k$ is defined similarly as ${\theta }_2^{\ast }$ [4]. The function on the left hand side of the above inequality is called the sensitivity function. Of course if the trial value of k is indeed the number of support points of the optimal discrimination design, the equivalence theorem holds and we have ${\xi }_T^k={\xi }_T$ and ${\theta }_2^k={\theta }_2^{\ast }$. The theorem applies to continuous designs, but not to exact designs.

2.2. Semi-Infinite Programming

Hettich and Kortanek [23] and López and Still [28] provide surveys of the theory, applications and recent development of SIP methodology. Broadly speaking, the numerical methods employed to solve SIP problems fall into three classes: exchange methods, discretization based methods and local reduction based methods [22]. Here we use an exchange based procedure similar to the one proposed by Blankenship and Falk [8], and further expounded in žakovíc and Rustem [48] among others. To this end, consider the general minimax program formalization used by Rustem and Howe [37] and žakovíc and Rustem [48]:

$$\begin{array}{ll}\hfill \underset{y}{min}\underset{z}{max}& f(y,z)\hfill \\ \hfill \text{s.t.}& g_{l_1}(y,z)\le 0,l_1\in \{1,\dots ,N_I\}\hfill \\ \hfill & h_{l_2}(y,z)=0,l_2\in \{1,\dots ,N_E\}\hfill \\ \hfill & y\in \mathcal{Y},z\in \mathcal{Z},\hfill \end{array}$$ (5)

where y ∈ 𝒴 ⊂ ℝ^n_y are the outer problem decision variables and z ∈ 𝒵 ⊂ ℝ^n_z are the decision variables of the inner problem. The set 𝒴 ≡ {y : g_l1 (y, z( ≤ 0, h_l2 (y, z( = 0, l₁ ∈ {1, …, N_I}, l₂ ∈ {1, …, N_E}} encapsulates all constraints involving y and the set 𝒵 encapsulates all constraints involving z, with g_l1 (y, z( representing the inequality constraints and h_l2 (y, z( the equality constraints. Both 𝒴 and 𝒵 are compact sets, all the functions g_l1 (y, z( and h_l2 (y, z( are differentiable and 𝒵 is a set dependent on y. The function f (y, z( is assumed to be differentiable in y and z and convex as a function of the outer problem decision variables y. No assumptions relative to the convexity properties of f (y, z( with respect to inner level decision variables are considered. This formulation has an outer problem (i.e. the min problem) and an inner problem (i.e. the max problem) and we solve the minimax program in two phases, Phase 1 and Phase 2 iteratively, until a convergence condition is satisfied.

At the nth iteration, there exists τⁿ ∈ ℝ: max_z∈𝒵 f(y, z( ≤ τⁿ if and only if f (y, z( ≤ τⁿ, ∀z ∈ 𝒵. Accordingly, we may formulate an equivalent semi-infinite program using a relaxation procedure to find the solution of the minimax problem as follows [38]:

$$\begin{array}{ll}\hfill \underset{y\in \mathcal{Y},{\tau }^n\in [{\tau }^L,{\tau }^U]}{min}& {\tau }^n\hfill \\ \hfill \text{s.t.}& f(y,z)\le {\tau }^n\hfill \\ \hfill & g_{l_1}(y,z)\le 0,l_1\in \{1,\dots ,N_I\}\hfill \\ \hfill & h_{l_2}(y,z)=0,l_2\in \{1,\dots ,N_E\}\hfill \\ \hfill & y\in \mathcal{Y},z\in \mathcal{Z}.\hfill \end{array}$$ (6)

Here τ^L and τ^U are finite values bounding τⁿ and since they are unknown, we may consider τ^L equal to a finite large negative value and τ^U equal to a finite large positive constant. The problem (6) involves a finite number of variables and an infinite number of constraints as a result of the dependency of 𝒵(y(.

The reformulation of problem (6) to an equivalent problem with a finite number of constraints requires that we replace 𝒵 with a discrete set. At the first iteration, we denote this set by 𝒵¹ = {z₀} where z₀ is feasible solution of the inner program prescribed in Section 3. At the nth iteration, this set is 𝒵ⁿ and has n elements originating in previous iterations. At the next iteration, this set becomes 𝒵ⁿ⁺¹ with n + 1 elements formed by augmenting 𝒵ⁿ with a solution for the Phase 2 problem (9), denoted by z_n, following the rule:

$${\mathcal{Z}}^{n+1}={\mathcal{Z}}^n\cup \{z_n\}.$$ (7)

The Phase 1 program, denoted as 𝒫_1,A, to solve is therefore:

$$\begin{array}{ll}\hfill \underset{y\in \mathcal{Y},{\tau }^n\in [{\tau }^L,{\tau }^U]}{min}& {\tau }^n\hfill \\ \hfill \text{s.t.}& f(y,z)\le {\tau }^n\hfill \\ \hfill & g_{l_1}(y,z)\le 0,l_1\in \{1,\dots ,N_I\}\hfill \\ \hfill & h_{l_2}(y,z)=0,l_2\in \{1,\dots ,N_E\}\hfill \\ \hfill & y\in \mathcal{Y},z\in {\mathcal{Z}}^n.\hfill \end{array}$$ (8)

The problem 𝒫_1,A solves the outer level of (5) and each solution y minimizes the objective function for a set of discrete points z ∈ 𝒵ⁿ. Afterwards, we fix y and solve the following program corresponding to the inner program of the problem (5), denoted by 𝒫_1,B:

$$\begin{array}{ll}\hfill {\zeta }^n=\underset{z\in \mathcal{Z}}{max}& f(y,z)\hfill \\ \hfill \text{s.t.}& g_{l_1}(y,z)\le 0,l_1\in \{1,\dots ,N_I\}\hfill \\ \hfill & h_{l_2}(y,z)=0,l_2\in \{1,\dots ,N_E\}\hfill \\ \hfill & y\text{fixed},z\in \mathcal{Z}.\hfill \end{array}$$ (9)

The solution of (9), z_n, with the subscript n representing the iteration counter, are stationary/Karush–Kuhn–Tucker (KKT) points of the inner problem and are appended to the set 𝒵ⁿ employing (7). Then we repeat the cycle and keep iterating between the outer problem corresponding to Phase 1 and the inner problem, corresponding to Phase 2, until convergence occurs. The discrete set 𝒵ⁿ contains the accumulating successive KKT points of the inner program that produce successively tighter relaxations of (8).

We observe that the number of constraints f (y, z( ≤ τⁿ for the problem (8) increases by one per iteration as a result of the increase in the number of elements forming the set of discrete points 𝒵ⁿ. Solving problem 𝒫_1,A provides a global lower bound to the minimax problem and solving problem 𝒫_1,B produces a local upper bound (obtained for a particular point y). Therefore, τⁿ ≥ τⁿ⁻¹ but no conclusion can be drawn for ζⁿ in successive iterations, ζⁿ being the optimum of problem 𝒫_1,B. The convergence test checks the condition |(ζⁿ − τⁿ(/τⁿ| ≤ ε₁, where ε₁ is a positive small constant provided by the user to assess the relative error. When the condition is satisfied the solution has been found. Theoretical results prove that the procedure described above converges in a finite number of iterations for ε₁-optimal solutions [8,27].

Here we assume all constraints in the problem (5) are decoupled. This assumption is reasonable since in the optimal design problem the constraints are functions of the regressors or of the parameters and not on both types of variables. Strategies for this more complicated case are provided by Polak [32, Ch. 3], Mitsos et al. [31] and Tsoukalas et al. [40].

3. Algorithms

In this section we describe the SIP algorithms for finding T-optimal designs. This approach assumes that we want to find a k-point T-optimal design where k is pre-specified. In our algorithm k is initialized to the number of parameters in the problem plus one. If at convergence, the T-optimal design found by SIP is not optimal according to the equivalence theorem, we will repeat the search among designs with k + 1 points. Our experience is that usually a couple of such iterations will produce the SIP-generated T-optimal design that is optimal among all designs on the design space.

3.1. SIP formulation for T-optimal designs

In this section, we apply the general techniques in Section 2 to solve Problem (3) by finding the optimal discrimination design supported at k points. Accordingly we include a superscript k in the variables in the mathematical codes below. At the nth iteration of the SIP-based procedure, the generated design ξ^k,n has $x_i^{k,n}$ as its ith support point with corresponding weight $w_i^{k,n}$, i = 1, …, k, and they are found by solving the preceding optimization problem. This formulation corresponds to a direct application of the Phase 1 problem (8):

$$\begin{array}{ll}\hfill \underset{{\xi }^{k,n}\in \mathit{\Xi },{\tau }^{k,n}\in [{\tau }^L,{\tau }^U]}{min}& {\tau }^{k,n}\hfill \\ \hfill \text{s.t.}& -\sum _{i=1}^k{[{\eta }_t(x_i^{k,n})-{\eta }_2(x_i^{k,n},{\theta }_2^k)]}^2{w}_i^{k,n}\le {\tau }^{k,n}\hfill \\ \hfill & \sum _{i=1}^k{w}_i^{k,n}=1\hfill \\ \hfill & {\theta }_2^k\in {\mathit{\Theta }}_2^{k,n}.\hfill \end{array}$$ (10)

For a fixed design ξ^k,n, the program 𝒫_1,B for Phase 2 is:

$${\zeta }^{k,n}=\underset{{\theta }_2\in {\mathit{\Theta }}_2}{max}-\sum _{i=1}^k{[{\eta }_t(x_i^{k,n})-{\eta }_2(x_i^{k,n},{\theta }_2)]}^2{w}_i^{k,n}.$$ (11)

Let ${\theta }_2^{k,n}$ solve problem (11) and let ${\mathit{\Theta }}_2^{k,n+1}={\mathit{\Theta }}_2^{k,n}\cup \{{\theta }_2^{k,n}\}$ be the set of stationary/KKT points. The iteration between problems ((10)–(11)) allows convergence to the solution of the minimax problem and determines the optimal design, ${\xi }_T^k$. When convergence occurs $\mathit{\Delta }({\xi }_T^k)={\zeta }^{k,n}$ (at ε₁-error optimality) and the parametric combination at which the design is least efficient is ${\theta }_2^{k,o}={\theta }_2^{k,n}$, the argument of problem (11), corresponding to the least favorable parametric configuration in Θ₂ found at the nth iteration.

The algorithm proposed requires an initial feasible instance of θ₂ employed to form the discrete set ${\mathit{\Theta }}_2^{k,1}$. We generate a feasible initial solution by solving the above program, denoted by ${\mathcal{P}}_0^k$. The formalization assumes that a feasible solution can be found when we maximize the sum of squares for the lack of fit. The design obtained is T-optimal in distinguishing models for the most efficient combination of parameters for model discrimination.

$$\begin{array}{ll}\hfill \underset{{\xi }^{k,0}\in \mathit{\Xi },{\theta }_2\in {\mathit{\Theta }}_2}{min}& -\sum _{i=1}^k{[{\eta }_t(x_i^{k,0})-{\eta }_2(x_i^{k,0},{\theta }_2)]}^2{w}_i^{k,0}\hfill \\ \hfill \text{s.t.}& \sum _{i=1}^k{w}_i^{k,0}=1.\hfill \end{array}$$ (12)

Here ${\theta }_2^{k,0}$ is the argument of the problem (12) and ${\mathit{\Theta }}_2^{k,1}=\{{\theta }_2^{k,0}\}$.

The above search for the T-optimal design using the SIP-based method is summarized in Algorithm 1 below. To initiate the algorithm, the following input parameters are required: the tolerance level for testing convergence, the number of support points k, bounds for τ ^k,n which may be large negative and positive constants, bounds for 𝒳 and bounds for the parameters included in the “test model”. The algorithm then iterates to find ${\xi }_T^k$, the optimal discrimination design supported at k points, the least favorable parametric configuration vector of parameters, ${\theta }_2^{k,o}$ and the minimal value of the T-optimality criterion, $\mathit{\Delta }({\xi }_T^k)$.

Algorithm 1.

Algorithm to find a T-optimal design for k support points.

procedure FindDesign(ε1, k, τL, τU, θL, θU, ${\xi }_T^k,{\theta }_2^{k,o},\mathit{\Delta }({\xi }_T^k)$)
Solve ${\mathcal{P}}_0^k$	▷ Find an initial feasible solution
n ← 1
${\theta }_2^{k,0}\leftarrow arg({\mathcal{P}}_0^k),{\mathit{\Theta }}_2^{k,1}\leftarrow \{{\theta }_2^{k,0}\}$
while |(ζk,n − τk,n(/τk,n| ≤ ε1 do
Solve ${\mathcal{P}}_{1,A}^k$	▷ Solve the Phase 1 program
${\tau }^{k,n}\leftarrow arg({\mathcal{P}}_{1,A}^k)$
Fix ξk,n
Solve ${\mathcal{P}}_{1,B}^k$	▷ Solve the Phase 2 program
${\zeta }^{k,n}\leftarrow arg({\mathcal{P}}_{1,B}^k),{\theta }_2^{k,n}\leftarrow arg({\mathcal{P}}_{1,B}^k)$
${\mathit{\Theta }}_2^{k,n+1}\leftarrow {\mathit{\Theta }}_2^{k,n}\cup \{{\theta }_2^{k,n}\}$	▷ Update the set of KKT points
n ← n + 1	▷ Update the iteration counter
end while
${\xi }_T^k\leftarrow {\xi }^{k,n},{\theta }_2^{k,o}\leftarrow {\theta }_2^{k,n},\mathit{\Delta }({\xi }_T^k)\leftarrow {\zeta }^{k,n}$
end procedure

3.2. Checking the T-optimality of the k-support point design

The algorithm to find k-point T-optimal designs was discussed in Section 3.1. The task now is to have an automated process to check whether the k-point design obtained by solving the minimax program (3) is globally optimal among all designs on the design space. This task is addressed by iteratively solving a new nonlinear program ${\mathcal{P}}_2^k$. If the design space is one-dimensional, a univariate graphical plot will suffice to ascertain whether the equivalence theorem is satisfied. When the design space is multi-dimensional, with each dimension representing a different regressor, this task is harder because it is not easy to visually ascertain whether a high-dimensional plot satisfies the conditions of the equivalence theorem. Our procedure conducts a numerical search over 𝒳 automatically and so facilitates the decision making process whether the T-optimal design is found.

In practice, we start with k equal to p₂ + 1 and increase it by one sequentially until the SIP-generated T-optimal design satisfies the conditions in the equivalence theorem. Problem ${\mathcal{P}}_2^k$ below finds the maximum value of the sum of squares for the lack of fit of the models in 𝒳 for the k-point design ${\xi }_T^k$, and the optimal singleton parametric combination ${\theta }_2^{k,o}$:

$${\alpha }^k=\underset{x\in \mathcal{X}}{max}{[{\eta }_t(x)-{\eta }_2(x,{\theta }_2^{k,o})]}^2+\mathit{\Delta }({\xi }_T^k).$$ (13)

The equivalence theorem is validated if and only if α^k ≤ ε₂, ε₂ being a small positive constant supplied by the user. Algorithm 2 presents the complete procedure, that finds the T-optimal design and automates the search for the optimal number of support points.

Algorithm 2.

Algorithm to find a T-optimal design

procedure OptimalTDesign(ε2, p2, ε1, τL, τU, θL, θU, ${\xi }_T^k,{\theta }_2^{k,o},\mathit{\Delta }({\xi }_T^k)$)
k ← p2 + 1	▷ Initialization of k
Solve FindDesign(ε1, k, τL, τU, θL, θU, ${\xi }_T^k,{\theta }_2^{k,o},\mathit{\Delta }({\xi }_T^k)$)
Solve ${\mathcal{P}}_2^k$	▷ Check T-optimality for k = p2 + 1
${\alpha }^k\leftarrow arg({\mathcal{P}}_2^k)$
while αk ≥ ε2 do
k ← k + 1
Solve FindDesign(ε1, k, τL, τU, θL, θU, ${\xi }_T^k,{\theta }_2^{k,o},\mathit{\Delta }({\xi }_T^k)$)
Solve ${\mathcal{P}}_2^k$	▷ Solve the NLP problem to check T-optimality
${\alpha }^k\leftarrow arg({\mathcal{P}}_2^k)$
end while
end procedure

Another indication that k is well chosen is that when the number of support points is greater than that for the optimal design there are one or more points with null weights. However, this indication can only be found if we solve the problem for such a k.

To implement the Semi-Infinite Programming based algorithm we require global NLP solvers to guarantee the global optimality of the solutions of problems ((10)–(13)). Here one uses the solver OQNLP from the GAMS package, which is formed by a general modeling language and a battery of mathematical programming solvers for several types of problems [19]. OQNLP is a multistart heuristic algorithm designed to find global optima of nonlinear programs. The algorithm calls a NLP solver from multiple starting points, saves all the feasible solutions found and picks the best as the optimum of the problem [43]. The starting points are computed with a random sampling driver that uses normal independent probability distribution functions for each decision variable. OQNLP does not guarantee that the final solution is a global optimum but it has been successfully tested on a large set of problems. To build the initial sampling points the variables need to be bounded, which we have since the design space and the region of plausible values are all compact by assumption.

The NLP solver called by OQNLP is CONOPT, which in turn uses the Generalized Reduced Gradient (GRG) algorithm [13]. The maximum number of starting points allowed is set to 3000 and the procedure terminates when 100 consecutive NLP solver calls result in a only fractional improvement of less than 10⁻⁴ in the criterion value. In all our problems the absolute and relative tolerances of the solvers were all respectively set equal to 10⁻⁸ and 10⁻⁷. The tolerances ε₁ and ε₂ required by Algorithms 1 and 2 were both set equal to 10⁻⁵. In all cases the bounds for τ^k,n were set to −10⁴ and +10⁴.

4. Results

We now apply the SIP-based algorithm to find the optimal discrimination design for discriminating among models. The selected applications are from the literature where we can compare results and verify the usefulness of the SIP-approach for finding T-optimal designs.

In Section 4.1 we first consider a benchmark problem to test our proposed SIP-based approach. Accordingly we provide more details, including analysis of the convergence properties of the SIP algorithm. In Section 4.2, we consider 8 optimal discrimination design problems; the first 4 concern univariate polynomial models all defined on a common design space 𝒳 = [−1, +1] and the next 3 concern nonlinear models with one or two regressors. In the last problem, case (8) has 3 models for discrimination by design and so the optimization setup is more complicated.

Table 1 displays the T-optimal designs for cases (1)–(7) and Table 2 displays the T-optimal design for case (8). For each case, we list the optimal discrimination design, the least favorable parametric configuration vector of parameters, ${\theta }_2^{k,o}$, that gives the largest lack of fit sum of squares, the optimum value for the design criterion and the CPU time in seconds (secs) required to solve the optimization problem. All results were obtained using a 12 GB RAM Intel Core i7 machine running 64 bits Windows 7 operating system with 2.80 GHz. The solvers used to handle all NLP programs are from GAMS package, version 23.5. Throughout, we let η_t denote the mean function of the “true model” and let η₁ and η₂ denote the mean functions of the test or alternative models. We assume the p₂ × 1 vector of unknown parameters θ₂ in η₂ is from a user-selected set Θ₂ which we assume is a Cartesian product of the parameter space for each component.

Table 1.

T-optimal designs for Cases (1)–(7) and their characteristics.

Case	k	T-optimal design	θk,o	optimum value	CPU (s)
1	2	$\left(\begin{array}{cc}-0.5000& 1.0000\\ 0.5003& 0.4997\end{array}\right)$	{1.8742}	−1.265625	7.526
2	3	$\left(\begin{array}{ccc}-1.0000& 0.0000& 1.0000\\ 0.2500& 0.5000& 0.2500\end{array}\right)$	{1.5000, 1.0000}	−0.250000	207.324
3	5	$\left(\begin{array}{ccccc}-1.0000& -0.5432& 0.1803& 0.7731& 1.0000\\ 0.0555& 0.1594& 0.2580& 0.3408& 0.1864\end{array}\right)$	{0.8936, 0.5416, 1.9550, 2.4591}	−0.022747	361.575
4	5	$\left(\begin{array}{ccccc}-1.0000& -0.8090& -0.3090& 0.3090& 0.8090\\ 0.2004& 0.3586& 0.2642& 0.1376& 0.0391\end{array}\right)$	{0.9997, 0.6882, 0.9989, 2.2480}	−0.003906	335.324
5	3	$\left(\begin{array}{ccc}0.9808& 3.6603& 5.0000\\ 0.4548& 0.3665& 0.1787\end{array}\right)$	{2.0837, 2.8740}	−0.231310	252.643
6	4	$\left(\begin{array}{cccc}3.0580& 5.4390& 30.0000& 30.0000\\ 0.0000& 11.6506& 22.7304& 0.0000\\ 0.2498& 0.4415& 0.2496& 0.0590\end{array}\right)$	{11.8718, 7.6432, 12.7019}	−0.533095	187.869
7	4	$\left(\begin{array}{cccc}1.8152& 4.0914& 30.0000& 30.0000\\ 0.0000& 4.1462& 0.0000& 10.1666\\ 0.0461& 0.5498& 0.0666& 0.3375\end{array}\right)$	{8.3470, 2.1013, 0.6554}	−0.867212	222.576

k – number of support points.

Table 2.

T-optimal design for Case (8) with its characteristics.

Case	k	T-optimal design	θk,o	Optimum value	CPU (s)
8	5	$\left(\begin{array}{ccccc}-1.0000& -0.7364& -0.0989& 0.6247& 1.0000\\ 0.2022& 0.3306& 0.2263& 0.1664& 0.0744\end{array}\right)$	$\left\{\begin{array}{c}1.0284,0.5634,-1.9201,\\ -0.8252,0.5930,1.8928,-0.1876\end{array}\right\}$	0.003195	804.695

k–number of support points.

4.1. Test case

We first consider the problem discussed in Atkinson and Fedorov [5] and Atkinson et al. [4] where

$${\eta }_t=4.5-1.5exp(x_1)-2.0exp(-x_1)$$ (14)

$$\begin{array}{l}{\eta }_2={\theta }_1+{\theta }_2{x}_1+{\theta }_3{x}_1^2\\ \mathcal{X}=[-1,+1],\mathit{\Theta }=[-10,4]\times [-10,4]\times [-10,4]\end{array}$$ (15)

and compare their T-optimal design with the one from our SIP based algorithm. The alternative model is clearly linear with respect to the parameters θ_i, i = {1, 2, 3} and nonlinear in the regressors. For this application, we set the initial number of support points to be 4 and find the optimal discrimination design among all 4-point designs. The SIP requires 28 iterations; Fig. 1 presents the evolution of the lower and upper bounds. The absolute error, defined in Algorithm 1 as |ζ^k,n − τ^k,n|, corresponds to the absolute difference between the upper and the lower bounds at the nth iteration. A value of magnitude 10⁻³ is reached in 9 iterations; the subsequent iterations are necessary to converge to a relative error, defined as |(ζ^k,n − τ^k,n(/τ^k,n|, of 10⁻⁵. It is noteworthy that the convergence condition used in Algorithm 1, which relies on the relative error, is responsible for an important fraction of the iterations. This trend cannot be generalized, but depends on the specific problem. The analysis of the behavior of the upper bound reveals that its value is close to 10⁻⁶ during the first 20 iterations, due to the ability of the inner program to determine a parametric combination that nearly produces a null lack of fit sum of squares for the design produced in Phase 1. Thereafter, the value of the upper bound decreases to the optimum value of −0.003906 (see Case 4 in Table 1). The behavior of the lower bound, always increasing, is in agreement with the theoretical result presented in Section 2.1. Another aspect to mention is that the algorithm for the T-optimal design ${\xi }_T^4$ converges in 20 iterations but the convergence of θ^4,o requires 8 additional iterations. This trend cannot be generalized to all problems, but in this case the least favorable parametric configuration is more difficult to find than the optimal design.

Fig. 1.

Evolution of the upper and lower bounds of the 4-point T-optimal design in case (1).

The optimal design found is:

$$\left(\begin{array}{cccc}-1.0000& -0.6693& 0.1438& 0.9570\\ 0.2536& 0.4250& 0.2497& 0.0718\end{array}\right)$$

for θ^4,o = {1.0288, 0.5550, −1.9292}. The optimum of the semi-infinite program is −0.001087, requiring a CPU time of 613.3970 s. The design obtained is in strong agreement with the results of Atkinson et al. [4], and satisfies the conditions for the equivalence theorem. Therefore the T-optimal design found, based on 4 support points, is globally optimal. Fig. 2 illustrates the lack of fit of the models for the best fitting combination of parameters, and the support points. Fig. 3 illustrates graphically the validation of the equivalence theorem.

Fig. 2.

Mean responses from the “True model” and the “test model” in case (1) with the vertical lines representing the support points of the T-optimal design.

Fig. 3.

Sensitivity function of the T-optimal design with the vertical lines representing the support points in case (1).

4.2. Additional test cases

We now consider 8 more cases for further testing the effectiveness of our proposed algorithms. The first 4 concern only univariate polynomial models all defined on a common design space 𝒳 ∈ [−1, +1]; the next 3 cases are nonlinear models with one or two regressors. Unlike the first 7 cases where only 2 models are involved, the last example, case (8), is to discriminate between 3 linear models, two of them considered as “test models”.

Case (1)

This example is taken from Atkinson and Fedorov [5] and Atkinson et al. [4], where interest is to find the T-optimal design to discriminate between a “true” quadratic model and a constant model. We have

$${\eta }_t=1+1x_1+1x_1^2$$ (16)

$$\begin{array}{l}{\eta }_2={\theta }_1\\ \mathcal{X}\in [-1,+1],\mathit{\Theta }\in [0,4].\end{array}$$ (17)

Case (2)

We now wish to find the T-optimal design to discriminate between the quadratic polynomial from a linear model on the same design space.

$${\eta }_t=1+1x_1+1x_1^2$$ (18)

$$\begin{array}{l}{\eta }_2={\theta }_1+{\theta }_2{x}_1\\ \mathcal{X}=[-1,+1],\mathit{\Theta }\in [0,4]\times [0,4].\end{array}$$ (19)

We note that with nested linear models that differ by a single parameter, the optimal discrimination design problem is the same as that for finding the best design to estimate that single parameter, i.e. we want to find the D_s-optimal design.

Case (3)

Dette et al. [11] considered finding T-optimal designs for discriminating among polynomials. The true model is a polynomial of degree n and the test models are polynomials of degree (n − 2(. A particular example of such a situation is when n = 5 with

$${\eta }_t=1+1x_1+1x_1^2+1x_1^3+1x_1^4+1x_1^5$$ (20)

$$\begin{array}{l}{\eta }_2={\theta }_1+{\theta }_2{x}_1+{\theta }_3{x}_1^2+{\theta }_4{x}_1^3\\ \mathcal{X}=[-1,+1],\mathit{\Theta }=[0,4]\times [0,4]\times [0,4]\times [0,4].\end{array}$$ (21)

Case (4)

A special instance of Case 3 was considered by Atkinson [1] and Dette et al. [11] where the coefficient of the term $x_1^{n-1}$ in the “true model” is 0. We have

$${\eta }_t=1+1x_1+1x_1^2+1x_1^3+0x_1^4+1x_1^5$$ (22)

$$\begin{array}{l}{\eta }_2={\theta }_1+{\theta }_2{x}_1+{\theta }_3{x}_1^2+{\theta }_4{x}_1^3\\ \mathcal{X}=[-1,+1],\mathit{\Theta }=[0,4]\times [0,4]\times [0,4]\times [0,4].\end{array}$$ (23)

The interesting feature of this problem is that the optimal design is not symmetric and not unique. The 5-point optimal design found by SIP is shown in Table 1. It has support points that range from −1 to 0.8090 and so is not supported at one of the extreme ends of the design space. The symmetrized design with a support range from −0.8090 to 1 is also optimal, as is any convex linear combination of the two designs. The sensitivity plot based on the equivalence theorem in Fig. 4 supports this claimed behavior, since any 5 of the 6 points with the same peaks can be selected as the equivalence theorem is satisfied not only at the points of the design, but also at x₁ = 1. This is therefore a potential support point of the optimal design. The convex linear combination of the two designs can be chosen to maximize some secondary design criterion.

Fig. 4.

Sensitivity function of the T-optimal design with the vertical lines representing the support points in case (4).

Case (5)

This example is taken from López-Fidalgo et al. [30] who were interested in finding the T-optimal design to discriminate between a Michaelis–Menten kinetic rate model augmented with a linear term and the simple Michaelis–Menten model.

$${\eta }_t=\frac{x_1}{1.0+x_1}+0.1x_1$$ (24)

$$\begin{array}{l}{\eta }_2=\frac{{\theta }_1{x}_1}{{\theta }_2+x_1}\\ \mathcal{X}=[{10}^{-5},5.0],\mathit{\Theta }=[{10}^{-3},30]\times [{10}^{-3},10].\end{array}$$ (25)

The test model is linear in θ₁ but nonlinear in θ₂.

Case (6)

This optimal discrimination design problem was discussed and addressed by Atkinson [2]. Here the interest is to find an optimal design to discriminate between two models for studying enzyme activity in the presence of chemical inhibitors. Both models are direct extensions of the Michaelis–Menten model discussed in Case (5). The null or “true model” assumes there is competitive inhibition and the “test model” assumes there is non-competitive inhibition.

$${\eta }_t=\frac{25.8x_1}{4.36(2.58+x_2)+2.58x_1}$$ (26)

$$\begin{array}{l}{\eta }_2=\frac{{\theta }_1{\theta }_3{x}_1}{({\theta }_2+x_1)({\theta }_3+x_2)}\\ \mathcal{X}=[{10}^{-5},30.0]\times [{10}^{-5},40.0],\mathit{\Theta }=[{10}^{-3},100]\times [{10}^{-3},18]\times [{10}^{-3},18].\end{array}$$ (27)

The test model is nonlinear in the parameters θ₂ and θ₃, but not in θ₁.

Case (7)

This problem was also addressed by Atkinson [2] and corresponds to the opposite of Case (6). It is assumed that the inhibition in the “true model” is non-competitive and the inhibition in the “test model” is competitive.

$${\eta }_t=\frac{51.6x_1}{(4.36+x_1)(5.16+x_2)}$$ (28)

$$\begin{array}{l}{\eta }_2=\frac{{\theta }_1{\theta }_3{x}_1}{{\theta }_2({\theta }_3+x_2)+{\theta }_3{x}_1}\\ \mathcal{X}=[{10}^{-5},30.0]\times [{10}^{-5},40.0],\mathit{\Theta }=[{10}^{-3},100]\times [{10}^{-3},18]\times [{10}^{-3},18].\end{array}$$ (29)

Table 1 presents the T-optimal designs found for the above cases. The results obtained agree with the literature, and in many cases the two sets of results are identical. This shows that the SIP-based strategy works for these examples and is flexible enough to handle polynomials and non-linear models in a model discrimination problem. Fig. 5 is the 2-dimensional plot of the sensitivity function for Case (6), and confirms that the 4-point design found by SIP is optimal for discrimination among all designs on the design space.

Fig. 5.

Sensitivity function for the T-optimal design in Case (6).

Case (8)

This case tests our algorithm in a higher-dimensional problem where we want to find the T-optimal design for discriminating among three rival models considered in Atkinson and Fedorov [6]. We have

$${\eta }_t=4.5-1.5exp(x)-2exp(-x)$$ (30)

$${\eta }_1={\theta }_1+{\theta }_{2}x+{\theta }_3{x}^2$$ (31)

$$\begin{array}{l}{\eta }_2={\theta }_4+{\theta }_{5}sin(0.5\pi x)+{\theta }_{6}cos(0.5\pi x)+{\theta }_{7}sin(\pi x)\\ \mathcal{X}=[-1,1],\mathit{\Theta }={[-10,4]}^7.\end{array}$$ (32)

The two alternative models have a total of 7 unknown parameters with θ = [θ₁, θ₂, θ₃, θ₄, θ₅, θ₆, θ₇]^T and there is only one regressor x ∈ X. Atkinson and Fedorov [6] formulated the problem as a minmaxmin program where the outer program is to find optimal weights for the lack of fit sum of squares associated with each alternative model. Here, we view the problem differently and interpret the weights as known probabilities, with each representing the probability that the alternative model is the true model. To fix ideas, we assume that the weights for this example are both equal to 0.5, but other values can also be used in the same way. The minimax program corresponding to the T-optimal design for this case is:

$$\begin{array}{ll}\hfill \underset{\xi \in \mathit{\Xi }}{min}\underset{\theta \in \mathit{\Theta }}{max}& -\sum _{i=1}^k(0.5{[{\eta }_t(x_i)-{\eta }_1(x_i,{\theta }_1,{\theta }_2,{\theta }_3)]}^2+0.5{[{\eta }_t(x_i)-{\eta }_2(x_i,{\theta }_4,{\theta }_5,{\theta }_6,{\theta }_7)]}^2)w_i\hfill \\ \hfill \text{s.t}& \sum _{i=1}^k{w}_i=1.\hfill \end{array}$$ (33)

The results in Table 2 are in close agreement with those from Atkinson and Fedorov [6] verifying once again that our SIP-based algorithm is able to solve more complex discrimination design problems.

There are two observations from the above examples. First, the CPU times required by our algorithm to find the optimal discrimination design can differ substantially from problem to problem. A main reason is that each problem requires differing amounts of time to solve the program ${\mathcal{P}}_{1,A}^k$ in each iteration. This problem is computationally intensive because it finds the globally optimal solutions by validating the conditions listed in Section 3 and in many cases, the algorithm requires between 3000 and 5000 calls of the NLP solver. Further this solution depends on the assumed dimension of the cartesian regions 𝒳 and Θ₂, and on the efficiency of the sampling driver used to generate initial solutions for the NLPs solved.

Second, the T-optimal designs for the first seven cases have k = p₂ + 1 support points. This number is that obtained on theoretical grounds by Dette and Titoff [12] for T-optimal designs when all models of interest are polynomials. Case (8) in Table 2 has 7 parameters but leads to an optimal discrimination design with 5 support points. This number also validates the theoretical result if we adopt the rule that when there are 2 or more test models, p₂ should refer to the number of parameters in the mean function of the model with the most parameters among the alternative models. In this example, the model with more parameters among the two models is η₂(x_i, θ₄, θ₅, θ₆, θ₇( with 4 parameters, resulting in the T-optimal design having 5 points. The implication from our numerical results suggest that the theoretical result from Dette and Titoff [12] on the number of support points in a T-optimal design may apply to a larger class of models than polynomials.

5. Conclusions

Traditional statistical methods of finding optimal designs, such as Wynn–Fedorov algorithms, can run into convergence problems when the optimization problem becomes complex as in minimax or maximin optimization. The goal of this paper was to explore alternative, and potentially more flexible and powerful, methods to find optimal designs for discriminating between two or more models. We first formulated the minimax optimization problem as one belonging to a class of non-smooth mathematical programs before applying Semi-Infinite Programming tools for numerical solution of the minimax optimization problem using an exchange based algorithm. To do this, the continuous domain of the parameters is discretized so that we can reformulate the original SIP problem with an infinite number of constraints as a relaxation program with a finite number of constraints.

The discretization points that populate the discrete set of parametric combinations are solutions of the inner program; the solution of the SIP is obtained through convergence of the solutions of the outer and the inner programs. Our approach includes an initial step to find a feasible initial parametric combination to start the iterative procedure. We automate the search for the global T-optimal design and confirm the optimality of the SIP-generated design using an equivalence theorem. If the SIP-generated design is not T-optimal, the next search will be among designs with one more support point. Our experience is that usually just a couple of such increases will suffice before the process finds the T-optimal design. All programs were solved using a stochastic global NLP solver.

We tested our SIP approach with a number of discrimination design problems for linear and nonlinear models. In all cases, our results from the SIP approach were in agreement with the results reported in the literature. The main advantages of our proposed approach are its flexibility and the relatively low computational resources required to find the T-optimal designs when compared with traditional methods. We are encouraged by SIP methodology and plan to test its ability to find optimal discrimination designs more broadly in more complicated problems. For example, a good test case is to find robust T-optimal designs where we do not require one of the two models to be fully known [44]. Mathematical programming tools such as SIP and Semi-definite programming have already been commonly and effectively used by engineers for decades but not by statisticians. We hope that our work here will encourage statisticians to explore such programming tools in their research work.