A design criterion for symmetric model discrimination based on flexible nominal sets

Abstract

Experimental design applications for discriminating between models have been hampered by the assumption to know beforehand which model is the true one, which is counter to the very aim of the experiment. Previous approaches to alleviate this requirement were either symmetrizations of asymmetric techniques, or Bayesian, minimax, and sequential approaches. Here we present a genuinely symmetric criterion based on a linearized distance between mean‐value surfaces and the newly introduced tool of flexible nominal sets. We demonstrate the computational efficiency of the approach using the proposed criterion and provide a Monte‐Carlo evaluation of its discrimination performance on the basis of the likelihood ratio. An application for a pair of competing models in enzyme kinetics is given.

1. INTRODUCTION

Besides optimization and parameter estimation, discrimination between rival models has always been an important objective of an experiment, and, therefore, of the optimization of experimental design. The crucial problem is that one typically cannot construct an optimal model‐discrimination design without already knowing which model is the true one, and what are the true values of its parameters. In this respect, the situation is analogous to the problem of optimal experimental design for parameter estimation in nonlinear statistical models (e.g., Pronzato & Pazman, ²⁰¹⁴), and many standard techniques can be used to tackle the dependence on the unknown characteristics: local, Bayesian, minimax, and sequential approaches, as well as their various combinations.

A big leap from initial ad hoc methods for model discrimination (see Hill, ¹⁹⁷⁸, for a review) was Atkinson and Fedorov (1975) who introduced T‐optimality, derived from the likelihood‐ratio test under the assumption that one model is true and its parameters are fixed at nominal values chosen by the experimenter. There, maximization of the noncentrality parameter is equivalent to maximizing the power of the likelihood‐ratio test for the least favorable parameter of the model assumed to be wrong. Thus, T‐optimality can be considered a combination of a localization and a minimax approach.

When the models are nested and (partly) linear, T‐optimality can be shown to be equivalent to $D_s$‐optimality for a single parameter that embodies the deviations from the smaller model (see, e.g., Dette & Titoff, ²⁰⁰⁹; Stigler, ¹⁹⁷¹). For this setting the optimal design questions are essentially solved and everything hinges on the asymmetric nature of the NP‐lemma with respect to the null‐ and alternative hypotheses. However, for a nonnested case the design problem itself is often inherently symmetric with respect to the exchangeability of the compared models and it is the purpose of the experiment to decide which of those two different models is true.

The aim of this paper is to solve the discrimination design problem in a symmetric way focusing on nonnested models. Thus, standard methods that are inherently asymmetric like T‐optimality, albeit being feasible, are not a natural choice. We further suppose that we do not use the full prior distribution of the unknown parameters of the models, which rules out Bayesian approaches such as Felsenstein (1992) and Tommasi and López‐Fidalgo (2010). Nevertheless, as we will make more precise in the next section, we will utilize what can be perceived as a specific kind of prior knowledge about the unknown parameters, extending the approach of local optimality. Our goal is to provide a lean, computationally efficient and scalable method as opposed to the heavy machinery recently employed in the computational statistics literature, for example, Hainy, Price, Restif, and Drovandi (2018). Furthermore, we strive for practical simplicity, which at first prohibits sequential (see Buzzi‐Ferraris & Forzatti, ¹⁹⁸³; Müller & Ponce De Leon, ¹⁹⁹⁶; Schwaab et al., ²⁰⁰⁶) or sequentially generated (see Vajjah & Duffull, ²⁰¹²) designs.

A standard solution to the symmetric discrimination design problem is to employ symmetrizations of asymmetric criteria such as compound T‐optimality, which usually depend on some weighting chosen by the experimenter. Also the minimax strategy recently presented in Tommasi, Martín‐Martín, and López‐Fidalgo (2016) is essentially a symmetrization. Moreover, the usual minimax approaches lead to designs that completely depend on the possibly unrealistic extreme values of the parameter space and their calculation again demands enormous computational effort.

As the closest in spirit to our approach could be considered a proposal for linear models in section 4.4 of Atkinson and Fedorov (1975) and its extension in Fedorov and Khabarov (1986) which, however, was not taken up by the literature. The probable reason is that it involves some rather arbitrary restrictions on the parameters as well as taking an artificial lower bound to convert it into a computationally feasible optimization problem.

For expositional purposes we will now restrict ourselves to a rather specific design task but will discuss possible extensions at the end of the paper.

Let $\mathfrak{X}\ne \varnothing$ be a finite design space and let $\mathcal{D}$ be a design on $\mathfrak{X}$, that is, a vector of design points $x_1,\dots ,x_n\in \mathfrak{X}$, where n is the chosen size of the experiment. Hence, in the terminology of the theory of optimal experimental design, we will work with exact designs. We will consider discrimination between a pair of nonlinear regression models

$$\begin{array}{ccc}\hfill y_i& =& {\eta }_0({\theta }_0,x_i)+{\epsilon }_i,i=1,\dots ,n,\text{and}\hfill \\ \hfill y_i& =& {\eta }_1({\theta }_1,x_i)+{\epsilon }_i,i=1,\dots ,n,\hfill \end{array}$$

where $y_1,\dots ,y_n$ are observations, ${\eta }_0:{\mathrm{\Theta }}_0\times \mathfrak{X}\to \mathbb{R}$, ${\eta }_1:{\mathrm{\Theta }}_1\times \mathfrak{X}\to \mathbb{R}$ are the mean‐value functions, ${\mathrm{\Theta }}_0\subseteq {\mathbb{R}}^{m_0}$, ${\mathrm{\Theta }}_1\subseteq {\mathbb{R}}^{m_1}$ are parameter spaces with nonempty interiors int(Θ₀), int(Θ₁), and ${\epsilon }_1,\dots ,{\epsilon }_n$ are unobservable random errors. For both $k=0,1$ and any $x\in \mathfrak{X}$, we will assume that the functions ${\eta }_k(·,x)$ are differentiable on $\mathrm{int}({\mathrm{\Theta }}_k)$; the gradient of ${\eta }_k(·,x)$ in ${\theta }_k\in \mathrm{int}({\mathrm{\Theta }}_k)$ will be denoted by $\nabla {\eta }_k({\theta }_k,x)$. Our principal assumption is that one of the models is true but we do not know which, that is, for $k=0$ or for $k=1$ there exists ${\overline{\theta }}_k\in {\mathrm{\Theta }}_k$ such that $y_i={\eta }_k({\overline{\theta }}_k,x_i)+{\epsilon }_i$.

Let the random errors be i.i.d. $N(0,{\sigma }^2)$, where ${\sigma }^2\in (0,\infty )$. The assumption of the same variances of the errors for both models is plausible if, for instance, the errors are due to the measurement device and hence do not significantly depend on the value being measured. The situation with different error variances requires a more elaborate approach, compared with Fedorov and Pázman (1968).

Eventually we are aiming not just at achieving some high design efficiencies with respect to our newly proposed criterion, but also want to test its usefulness in concrete discrimination experiments, that is, the probability that using our design we arrive at the correct decision about which model is the true one. So, to justify our approach numerically, we require a model‐discrimination rule that will be used after all observations based on the design $\mathcal{D}$ are collected for evaluational purposes.

The choice of the best discrimination rule based on the observations is generally a nontrivial problem. However, it is natural to compute the maximum likelihood estimates ${\widehat{\theta }}_0$ and ${\widehat{\theta }}_1$ of the parameters under the assumption of the first and the second model, respectively, and then base the decision on whether

$$\frac{L({\widehat{\theta }}_0|{(y_i)}_{i=1}^n)}{L({\widehat{\theta }}_1|{(y_i)}_{i=1}^n)}<>1,$$ (1)

that is, the likelihood ratio being smaller or greater than 1, or perhaps more simply whether $\log L({\widehat{\theta }}_0)-\log L({\widehat{\theta }}_1)<>0$. Under the normality, homoskedasticity, and independence assumptions, this decision is equivalent to a decision based on the proximity of the vector ${(y_i)}_{i=1}^n$ of observations to the vectors of estimated mean values ${({\eta }_0({\widehat{\theta }}_0,x_i))}_{i=1}^n$ and ${({\eta }_1({\widehat{\theta }}_1,x_i))}_{i=1}^n$.

For the case $m_0\ne m_1$ to counterbalance favoring models with greater number of parameters, Cox (2013) recommends instead the use of $L({\widehat{\theta }}_0)/L({\widehat{\theta }}_1){(e^{m_1}/e^{m_0})}^{n/\stackrel{\sim }{n}}$, which corresponds to the Bayesian information criterion; see Schwarz (1978). Here $\stackrel{\sim }{n}$ corresponds to the number of observations in a real or fictitious prior experiment. For the sake of simplicity however, we will now restrict ourselves to the case of $m:=m_0=m_1$. Note that for the evaluational purposes we are taking a purely model selection based standpoint. More sophisticated testing procedures for instance allowing both models to be rejected based on the pioneering work of Cox (1961) are reviewed and outlined in Pesaran and Weeks (2007).

Let $x_1,\dots ,x_n\in \mathfrak{X}$ and let $\mathcal{D}=(x_1,\dots ,x_n)$ be the design used for the collection of data prior to the decision, and assume that model η₀ is true, with the corresponding parameter value ${\overline{\theta }}_0$. Note that this comes without loss of generality and symmetry as we can equivalently assume model η₁ to be true. Then, the probability of the correct decision based on the likelihood ratio is equal to

$$P\left[\underset{{\theta }_0\in {\mathrm{\Theta }}_0}{\min }\sum _{i=1}^n{({\eta }_0({\theta }_0,x_i)-y_i)}^2\le \underset{{\theta }_1\in {\mathrm{\Theta }}_1}{\min }\sum _{i=1}^n{({\eta }_1({\theta }_1,x_i)-y_i)}^2\right],$$ (2)

where ${(y_i)}_{i=1}^n$ follows the normal distribution with mean ${({\eta }_0({\overline{\theta }}_0,x_i))}_{i=1}^n$ and covariance ${\sigma }^2{I}_n$.

Clearly, probability (2) depends on the true model, the unknown true parameter, and also on the unknown variance of errors. Even if these parameters were known, the probability of the correct classification would be very difficult to compute for a given design because this requires a combination of high‐dimensional integration and multivariate nonconvex optimization. Therefore, it is practically impossible to directly optimize the design based on formula (2). However, we can simplify the problem by constructing a lower bound on (2) which does not depend on unknown parameters and is relatively much simpler to maximize with respect to the choice of the design. The bound based on the distance $d(E_0,E_1)$, where $E_j$ is the set of all possible mean values of the observations under the model j, $j=0,1$, and d(., .) denotes the infimum distance between all pairs of elements of two sets, is developed as follows.

Consider a fixed experimental design $(x_1,\dots ,x_n)$, and denote $y:={(y_i)}_{i=1}^n$, ${\eta }_j({\theta }_j):={({\eta }_j({\theta }_j,x_i))}_{i=1}^n$ for $j=0,1$. Note that we can express (2) as $P[d(E_0,y)\le d(E_1,y)]$. Now, let $R=\parallel \epsilon \parallel$, where $\epsilon =y-{\eta }_0({\overline{\theta }}_0)$, be the norm of the vector of errors. Assuming $R\le d(E_0,E_1)/2$ we obtain

$$\begin{array}{ccc}& & d(E_0,E_1)\le d({\eta }_0({\widehat{\theta }}_0),{\eta }_1({\widehat{\theta }}_1))\le d(y,{\eta }_0({\widehat{\theta }}_0))+d(y,{\eta }_1({\widehat{\theta }}_1))\hfill \\ & & \le d(y,{\eta }_0({\overline{\theta }}_0))+d(y,{\eta }_1({\widehat{\theta }}_1))=R+d(y,{\eta }_1({\widehat{\theta }}_1))\le d(E_0,E_1)/2+d(y,{\eta }_1({\widehat{\theta }}_1)),\hfill \end{array}$$

which implies $d(E_0,E_1)/2\le d(y,{\eta }_1({\widehat{\theta }}_1))$ and consequently

$$d(E_0,y)=d(y,{\eta }_0({\widehat{\theta }}_0))\le d(y,{\eta }_0({\overline{\theta }}_0))=R\le d(E_0,E_1)/2\le d(y,{\eta }_1({\widehat{\theta }}_1))=d(E_1,y).$$

Thus, the event $[R\le d(E_0,E_1)/2]$ implies the event $[d(E_0,y)\le d(E_1,y)]$, that is, (2) can be bounded from below by

$$P\left[R\le d(E_0,E_1)/2\right].$$ (3)

To make (2) as high as possible, it makes sense to maximize (3), that is, maximize $d(E_0,E_1)$, which depends on the underlying experimental design. Although this maximization is much simpler than maximizing (2) directly, it still generally requires nonconvex multidimensional optimization at each iteration of the maximization procedure, which is impractical for computing exact optimal designs. A realistic approach must be numerically feasible and address the problem of the dependence of the design on unknown true model parameters, which we will achieve by rapidly computable approximation of $d(E_0,E_1)$ through linearization, as will be explained in the following section.

1.1. Example 1: A motivating example

Let ${\eta }_0({\theta }_0,x)={\theta }_{0}x$ and ${\eta }_1({\theta }_1,x)=e^{{\theta }_{1}x}$. Furthermore, for the moment we assume just two observations $y_1,y_2$ at fixed design points $x_1=-1$ and $x_2=1$, respectively. In this case evidently ${\widehat{\theta }}_0=\frac{y_2-y_1}{2}$ and ${\widehat{\theta }}_1$ is the solution of $2e^{-\theta }(y_1-e^{-\theta })-2e^{\theta }(y_2-e^{\theta })=0$, which for $-2\le y_1\le 2$ is the log root of the polynomial ${\gamma }^4-{\gamma }^3{y}_2+\gamma y_1-1$. Figure 1 displays the log‐likelihood‐ratio contours for the original and linearized models and it is obvious that the former are nonconvex and complex while the latter are much simpler, convex, and do approximate fairly well for a wide range of responses. Note that while this example is for a fixed design it motivates why the linearizations can serve as the cornerstones of our design method as will become clearer in the following sections.

Figure 1.

Left panel: contour plot of $\log L({\widehat{\theta }}_0)-\log L({\widehat{\theta }}_1)$ for Example 1, solid line corresponds to 0, horizontal y ₁, vertical y ₂; right panel: corresponding contour plot for the model η₁ linearized at ${\stackrel{\sim }{\theta }}_1=1$

2. THE LINEARIZED DISTANCE CRITERION

We suggest an extension of the idea of local optimality used for nonlinear experimental design. Let ${\stackrel{\sim }{\theta }}_0\in \mathrm{int}({\mathrm{\Theta }}_0)$ and ${\stackrel{\sim }{\theta }}_1\in \mathrm{int}({\mathrm{\Theta }}_1)$ be nominal parameter values, which satisfy the basic discriminability condition ${\eta }_0({\stackrel{\sim }{\theta }}_0,x)\ne {\eta }_1({\stackrel{\sim }{\theta }}_1,x)$ for some $x\in \mathfrak{X}$. Let us introduce regions ${\stackrel{\sim }{\mathrm{\Theta }}}_0\subseteq \mathrm{int}({\mathrm{\Theta }}_0)\subseteq {\mathbb{R}}^m$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1\subseteq \mathrm{int}({\mathrm{\Theta }}_1)\subseteq {\mathbb{R}}^m$ containing ${\stackrel{\sim }{\theta }}_0$ and ${\stackrel{\sim }{\theta }}_1$; we will consequently call ${\stackrel{\sim }{\mathrm{\Theta }}}_0$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1$ flexible nominal sets. It is evident that optimal designs depend on the parameter spaces in the same way as on our flexible nominal sets (cf. Dette, Melas, & Shpilev, ²⁰¹³), but the latter will not be considered fixed like the parameter spaces Θ₀ and Θ₁. A novelty of our procedure is that we use these sets as a tuning device.

Let $\mathcal{D}=(x_1,\dots ,x_n)$ be a design. Let us perform the following particular linearization of Model ${\eta }_{k=0,1}$ in ${\stackrel{\sim }{\theta }}_k$:

$${(y_i)}_{i=1}^n\approx {\mathbf{F}}_k(\mathcal{D}){\theta }_k+{\mathbf{a}}_k(\mathcal{D})+\epsilon ,$$

where ${\mathbf{F}}_k(\mathcal{D})$ is the $n\times m$ matrix given by

$${\mathbf{F}}_k(\mathcal{D})={\left(\nabla {\eta }_k({\stackrel{\sim }{\theta }}_k,x_1),\dots ,\nabla {\eta }_k({\stackrel{\sim }{\theta }}_k,x_n)\right)}^T,$$

${\mathbf{a}}_k(\mathcal{D})$ is the n‐dimensional vector

$${\mathbf{a}}_k(\mathcal{D})={({\eta }_k({\stackrel{\sim }{\theta }}_k,x_i))}_{i=1}^n-{\mathbf{F}}_k(\mathcal{D}){\stackrel{\sim }{\theta }}_k,$$

and $\epsilon ={({\epsilon }_1,\dots ,{\epsilon }_n)}^T$ is a vector of independent $N(0,{\sigma }^2)$ errors.

Note that for the proposed method the vector ${\mathbf{a}}_k(\mathcal{D})$ plays an important role and, although it is known, we cannot subtract it from the vector of observations, as is usual when we linearize a single nonlinear regression model. However, if ${\eta }_k$ corresponds to the standard linear model then ${\mathbf{a}}_k(\mathcal{D})={\mathbf{0}}_n$ for any $\mathcal{D}$.

2.1. Definition of the δ criterion

Consider the design criterion

$$\delta (\mathcal{D})=\underset{{\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0,{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1}{inf}\delta (\mathcal{D}|{\theta }_0,{\theta }_1),\text{where}$$ (4)

$$\delta (\mathcal{D}|{\theta }_0,{\theta }_1)=‖{\mathbf{a}}_0(\mathcal{D})+{\mathbf{F}}_0(\mathcal{D}){\theta }_0-\{{\mathbf{a}}_1(\mathcal{D})+{\mathbf{F}}_1(\mathcal{D}){\theta }_1\}‖,$$ (5)

for ${\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0,{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1$. The criterion δ can be viewed as an approximation of the nearest distance d of the mean‐value surfaces of the models, in the neighborhoods of the vectors ${({\eta }_0({\stackrel{\sim }{\theta }}_0,x_i))}_{i=1}^n$ and ${({\eta }_1({\stackrel{\sim }{\theta }}_1,x_i))}_{i=1}^n$; see the illustrative Figure 2.

Figure 2.

Illustrative graph for the definition of $\delta (\mathcal{D})$ for two one‐parameter models (${\mathrm{\Theta }}_0,{\mathrm{\Theta }}_1\subseteq \mathbb{R}$) and a design of size two ($\mathcal{D}=(x_1,x_2)$). The line segments correspond to the sets $\{{\mathbf{a}}_0(\mathcal{D})+{\mathbf{F}}_0(\mathcal{D}){\theta }_0:{\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0\}$ and $\{{\mathbf{a}}_1(\mathcal{D})+{\mathbf{F}}_1(\mathcal{D}){\theta }_1:{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1\}$ for some flexible nominal sets ${\stackrel{\sim }{\mathrm{\Theta }}}_0$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1$

We will now express the δ‐criterion as a function of the design $\mathcal{D}=(x_1,\dots ,x_n)$ represented by the counting measure ξ on $\mathfrak{X}$ defined as

$$\xi (\{x\}):=\#\{i\in \{1,\dots ,n\}:x_i=x\},x\in \mathfrak{X},$$

where $\#$ means the size of a set. Let $\stackrel{\sim }{\theta }={({\stackrel{\sim }{\theta }}_0^T,{\stackrel{\sim }{\theta }}_1^T)}^T$. For all $x\in \mathfrak{X}$ let

$$\begin{array}{ccc}\hfill \mathrm{\Delta }\eta (\stackrel{\sim }{\theta },x)& :=& {\eta }_0({\stackrel{\sim }{\theta }}_0,x)-{\eta }_1({\stackrel{\sim }{\theta }}_1,x),\hfill \\ \hfill \nabla \eta (\stackrel{\sim }{\theta },x)& :=& {\left(\nabla {\eta }_0^T({\stackrel{\sim }{\theta }}_0,x),-\nabla {\eta }_1^T({\stackrel{\sim }{\theta }}_1,x)\right)}^T.\hfill \end{array}$$

For any ${\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0$, ${\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1$ and $\theta ={({\theta }_0^T,{\theta }_1^T)}^T$ we have

$$\begin{array}{ccc}\hfill {\delta }^2(\mathcal{D}|{\theta }_0,{\theta }_1)& =& {‖{\mathbf{a}}_0(\mathcal{D})+{\mathbf{F}}_0(\mathcal{D}){\theta }_0-\{{\mathbf{a}}_1(\mathcal{D})+{\mathbf{F}}_1(\mathcal{D}){\theta }_1\}‖}^2\hfill \\ & =& \sum _{i=1}^n{\left(\nabla {\eta }^T(\stackrel{\sim }{\theta },x_i)(\theta -\stackrel{\sim }{\theta })+\mathrm{\Delta }\eta (\stackrel{\sim }{\theta },x_i)\right)}^2\hfill \\ & =& {\int }_{\mathfrak{X}}{\left(\nabla {\eta }^T(\stackrel{\sim }{\theta },x)(\theta -\stackrel{\sim }{\theta })+\mathrm{\Delta }\eta (\stackrel{\sim }{\theta },x)\right)}^2\mathrm{d}\xi (x).\hfill \end{array}$$ (6)

Therefore

$${\delta }^2(\mathcal{D}|{\theta }_0,{\theta }_1)={(\theta -\stackrel{\sim }{\theta })}^T\mathbf{M}(\xi ,\stackrel{\sim }{\theta })(\theta -\stackrel{\sim }{\theta })+2{\mathbf{b}}^T(\xi ,\stackrel{\sim }{\theta })(\theta -\stackrel{\sim }{\theta })+c(\xi ,\stackrel{\sim }{\theta }),$$ (7)

where

$$\mathbf{M}(\xi ,\stackrel{\sim }{\theta })={\int }_{\mathfrak{X}}\nabla \eta (\stackrel{\sim }{\theta },x)\nabla {\eta }^T(\stackrel{\sim }{\theta },x)\mathrm{d}\xi (x),$$ (8)

$$\mathbf{b}(\xi ,\stackrel{\sim }{\theta })={\int }_{\mathfrak{X}}\mathrm{\Delta }\eta (\stackrel{\sim }{\theta },x)\nabla \eta (\stackrel{\sim }{\theta },x)\mathrm{d}\xi (x),$$ (9)

$$c(\xi ,\stackrel{\sim }{\theta })={\int }_{\mathfrak{X}}{[\mathrm{\Delta }\eta (\stackrel{\sim }{\theta },x)]}^2\mathrm{d}\xi (x).$$ (10)

The matrix $\mathbf{M}(\xi ,\stackrel{\sim }{\theta })$ in Equations (7) and (8) can be recognized as the information matrix for the parameter θ in the linear regression model

$$\begin{array}{ccc}\hfill z_i& =& \nabla {\eta }^T(\stackrel{\sim }{\theta },x_i)\theta +{\epsilon }_i\hfill \\ & =& {[{\mathbf{F}}_0(\mathcal{D}),-{\mathbf{F}}_1(\mathcal{D})]}_{i·}\theta +{\epsilon }_i;i=1,\dots ,n,\hfill \end{array}$$ (11)

where ${[{\mathbf{F}}_0(\mathcal{D}),-{\mathbf{F}}_1(\mathcal{D})]}_{i·}$ is the ith row of the matrix $[{\mathbf{F}}_0(\mathcal{D}),-{\mathbf{F}}_1(\mathcal{D})]$, with parameter θ and independent, homoskedastic errors ${\epsilon }_1,\dots ,{\epsilon }_n$ with mean 0; we will call (11) a response difference model.

2.2. Computation of the δ criterion value for a fixed design

For a fixed design $\mathcal{D}$, expression (5) shows that ${\delta }^2(\mathcal{D}|\theta )$ is a quadratic function of $\theta ={({\theta }_0^T,{\theta }_1^T)}^T$. Moreover, both $\delta (\mathcal{D}|\theta )$ and ${\delta }^2(\mathcal{D}|\theta )$ are convex because they are compositions of an affine function of θ and convex functions $\parallel .\parallel$ and ${\parallel .\parallel }^2$, respectively. Clearly, if the flexible nominal sets are compact, convex, and polyhedral, optimization (4) can be efficiently performed by specialized solvers for linearly constrained quadratic programming.

Alternatively, we can view the computation of $\delta (\mathcal{D}|\theta )$ as follows. As

$${\delta }^2(\mathcal{D}|{\theta }_0,{\theta }_1)={‖\{{\mathbf{a}}_0(\mathcal{D})-{\mathbf{a}}_1(\mathcal{D})\}-[-{\mathbf{F}}_0(\mathcal{D}),{\mathbf{F}}_1(\mathcal{D})]\theta ‖}^2,$$

the minimization in (4) is equivalent to computing the minimum sum of squares for a least squares estimate of θ restricted to $\stackrel{\sim }{\mathrm{\Theta }}:={\stackrel{\sim }{\mathrm{\Theta }}}_0\times {\stackrel{\sim }{\mathrm{\Theta }}}_1$ in the response difference model with artificial observations

$${\stackrel{\sim }{z}}_i={\{{\mathbf{a}}_1(\mathcal{D})-{\mathbf{a}}_0(\mathcal{D})\}}_i,i=1,\dots ,n.$$

Thus, if ${\stackrel{\sim }{\mathrm{\Theta }}}_0={\stackrel{\sim }{\mathrm{\Theta }}}_1={\mathbb{R}}^m$, the infimum in (4) is attained, and it can be computed using the standard formulas of linear regression in the response difference model. If the flexible nominal sets are compact cuboids, (4) can be evaluated by the very rapid and stable method for bounded variable least squares implemented in the R package bvls; see Stark and Parker (1995) and Mullen (2013).

The following simple proposition collects the analytic properties of a natural analogue of δ defined on the linear vector space Ξ of all finite signed measures on $\mathfrak{X}$.

Proposition 2.1

For ${\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0$, ${\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1$ and a finite signed measure ξ on $\mathfrak{X}$ let ${\delta }_{app}^2(\xi |{\theta }_0,{\theta }_1)$ be defined via formula (6). Then, ${\delta }_{app}^2(·|{\theta }_0,{\theta }_1)$ is linear on Ξ. Moreover, let

$${\delta }_{app}^2(\xi ):=\underset{{\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0,{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1}{inf}{\delta }_{app}^2(\xi |{\theta }_0,{\theta }_1).$$

Then, ${\delta }_{app}^2$ is positive homogeneous and concave on Ξ.

Positive homogeneity of ${\delta }_{app}^2$ implies that an s‐fold replication of an exact design leads to an s‐fold increase of its δ² value. Consequently, a natural and statistically interpretable definition of relative δ‐efficiency of two designs ${\mathcal{D}}_1$ and ${\mathcal{D}}_2$ is given by ${\delta }^2({\mathcal{D}}_1)/{\delta }^2({\mathcal{D}}_2)$, provided that ${\delta }^2({\mathcal{D}}_2)>0$.

Let $\mathfrak{D}$ be the set of all n‐point designs. A design ${\mathcal{D}}^{\ast }\in \mathfrak{D}$ will be called δ‐optimal, if

$${\mathcal{D}}^{\ast }\in {\mathrm{argmax}}_{\mathcal{D}\in \mathfrak{D}}\delta (\mathcal{D}).$$

Note that the basic discriminability condition implies that if ${\stackrel{\sim }{\mathrm{\Theta }}}_0=\{{\stackrel{\sim }{\theta }}_0\}$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1=\{{\stackrel{\sim }{\theta }}_1\}$, then $\delta ({\mathcal{D}}^{\ast })$ is strictly positive. However, for larger flexible nominal sets it can happen that $\delta ({\mathcal{D}}^{\ast })=0$.

As the evaluation of the δ‐criterion is generally very rapid, the calculation of a δ‐optimal, or nearly δ‐optimal design is similar to that for standard design criteria. For instance, in small problems we can use complete‐enumeration and in larger problems we can employ an exchange heuristic, such as the KL exchange algorithm (see, e.g., Atkinson, Donev, & Tobias, ²⁰⁰⁷).

Note that the δ‐optimal designs depend not only on η₀, η₁, $\mathfrak{X}$, n, ${\stackrel{\sim }{\theta }}_0$, and ${\stackrel{\sim }{\theta }}_1$, but also on ${\stackrel{\sim }{\mathrm{\Theta }}}_0$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1$.

2.3. Parametrization of flexible nominal sets

For simplicity, we will focus on cuboid flexible nominal sets centered at the nominal parameter values. This choice can be justified by the results of Sidak (1967), in particular if we already have confidence intervals for individual parameters; see further discussion in Section 4. Specifically, we will employ the homogeneous dilations

$${\stackrel{\sim }{\mathrm{\Theta }}}_k^{(r)}:=r\left({\stackrel{\sim }{\mathrm{\Theta }}}_k^{(1)}-{\stackrel{\sim }{\theta }}_k\right)+{\stackrel{\sim }{\theta }}_k,r\in [0,\infty ),k=0,1,$$ (12)

${\stackrel{\sim }{\mathrm{\Theta }}}_0^{(\infty )}:={\mathbb{R}}^m$, ${\stackrel{\sim }{\mathrm{\Theta }}}_1^{(\infty )}:={\mathbb{R}}^m$, such that r can be considered a tuning (set) parameter governing the size of the flexible nominal sets. In (12), ${\stackrel{\sim }{\mathrm{\Theta }}}_0^{(1)}$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1^{(1)}$ are “unit” nondegenerate compact cuboids centered on respective nominal parameters. For any design $\mathcal{D}$ and $r\in [0,\infty ]$, we define

$${\delta }_r(\mathcal{D}):=\underset{{\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)},{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)}}{inf}\delta (\mathcal{D}|{\theta }_0,{\theta }_1).$$ (13)

Note that for our choice of flexible nominal sets the infimum in (13) is attained. The ${\delta }_r$‐optimal values of the problem will be denoted by

$$o(r):=\underset{\mathcal{D}\in \mathfrak{D}}{\max }{\delta }_r(\mathcal{D}).$$

Proposition 2.2

(a) Let $\mathcal{D}$ be a design. Functions ${\delta }_r^2(\mathcal{D})$, ${\delta }_r(\mathcal{D})$, $o^2(r)$, $o(r)$ are nonincreasing and convex in r on the entire interval [0, ∞]. (b) There exists $r^{\ast }<\infty$, such that for all $r\ge r^{\ast }$: (i) $o(r)=o(\infty )$; (ii) Any ${\delta }_{\infty }$‐optimal design is also a ${\delta }_r$‐optimal design.

(a) Let $\mathcal{D}$ be an n‐point design and let $0\le r_1\le r_2\in [0,\infty ]$.

Inequality ${\delta }_{r_1}^2(\mathcal{D})\ge {\delta }_{r_2}^2(\mathcal{D})$ follows from definitions (12) and (13), and inequality $o^2(r_1)\ge o^2(r_2)$ follows from the fact that a maximum of nonincreasing functions is a nonincreasing function. Monotonicity of ${\delta }_r(\mathcal{D})$ and $o(r)$ in r can be shown analogously.

To prove the convexity of ${\delta }_r^2(\mathcal{D})$ in r, let $\alpha \in (0,1)$ and let $r_{\alpha }=\alpha r_1+(1-\alpha )r_2$. For all $r\in [0,\infty ]$, let ${\widehat{\theta }}_r$ denote a minimizer of ${\delta }_r^2(\mathcal{D}|·)$ on ${\stackrel{\sim }{\mathrm{\Theta }}}^{(r)}:={\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)}\times {\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)}$. Convexity of ${\delta }^2(\mathcal{D}|\theta )$ in θ and a simple fact $\alpha {\widehat{\theta }}_{r_1}+(1-\alpha ){\widehat{\theta }}_{r_2}\in {\stackrel{\sim }{\mathrm{\Theta }}}^{(r_{\alpha })}$ yield

$$\begin{array}{c}\hfill \alpha {\delta }_{r_1}^2(\mathcal{D})+(1-\alpha ){\delta }_{r_2}^2(\mathcal{D})=\alpha {\delta }^2(\mathcal{D}|{\widehat{\theta }}_{r_1})+(1-\alpha ){\delta }^2(\mathcal{D}|{\widehat{\theta }}_{r_2})\\ \hfill \ge {\delta }^2(\mathcal{D}|\alpha {\widehat{\theta }}_{r_1}+(1-\alpha ){\widehat{\theta }}_{r_2})\ge {\delta }^2(\mathcal{D}|{\widehat{\theta }}_{r_{\alpha }})={\delta }_{r_{\alpha }}^2(\mathcal{D}),\end{array}$$

which proves that ${\delta }_r^2(\mathcal{D})$ is convex in r. The convexity of ${\delta }_r(\mathcal{D})$ in r can be shown analogously. The functions o ² and o, as pointwise maxima of a system of convex functions, are also convex.

(b) For any design $\mathcal{D}$ of size n, the function ${\delta }_{\infty }^2(\mathcal{D}|·)$ is nonnegative and quadratic on ${\mathbb{R}}^{2m}$, therefore its minimum is attained in some ${\theta }_{\mathcal{D}}\in {\mathbb{R}}^{2m}$. There is only a finite number of exact designs of size n, and ${\stackrel{\sim }{\mathrm{\Theta }}}^{(r)}{\uparrow }_r{\mathbb{R}}^{2m}$, which means that there exists $r^{\ast }<\infty$ such that ${\theta }_{\mathcal{D}}\in {\stackrel{\sim }{\mathrm{\Theta }}}^{(r^{\ast })}$ for all designs $\mathcal{D}$ of size n. Let $r\ge r^{\ast }$. We have

$$o(\infty )=\underset{D\in \mathfrak{D}}{\max }\underset{\theta \in {\mathbb{R}}^{2m}}{\min }{\delta }_{\infty }(\mathcal{D}|\theta )=\underset{D\in \mathfrak{D}}{\max }\underset{\theta \in {\stackrel{\sim }{\mathrm{\Theta }}}^{(r)}}{\min }\delta (\mathcal{D}|\theta )=\underset{D\in \mathfrak{D}}{\max }{\delta }_r(\mathcal{D})=o(r),$$

proving (i). Let ${\mathcal{D}}^{(\infty )}$ be any ${\delta }_{\infty }$‐optimal n‐trial design. The equality (i) and the fact that ${\delta }_r({\mathcal{D}}^{(\infty )})$ and $o(r)$ are nonincreasing with respect to r gives

$${\delta }_r({\mathcal{D}}^{(\infty )})\ge {\delta }_{\infty }({\mathcal{D}}^{(\infty )})=o(\infty )=o(r^{\ast })\ge o(r),$$

which proves (ii).$\square$

The second part of Proposition 2.2 implies the existence of a finite interval $[0,r^{\ast }]$ of relevant set parameters; increasing the set parameter beyond $r^{\ast }$ leaves the optimal designs as well as the optimal value of the δ‐criterion unchanged. We will call any such $r^{\ast }$ a set upper bound.

Algorithm 1 provides a simple iterative method of computing $r^{\ast }$. Our experience shows that it usually requires only a small number of recomputations of the ${\delta }_r$‐optimal design, even if $r_{ini}$ is small and q is close to 1, resulting in a good set upper bound $r^{\ast }$ (see the metacode of Algorithm 1 for details). Due to the high speed and stability of the computation of the values of ${\delta }_r$ for candidate designs, it is possible to use an adaptation of the standard KL exchange heuristic to compute the input value $o(\infty )$, as well as to obtain ${\delta }_r$‐optimal designs in steps 2 and 9 of the algorithm itself.

Algorithm 1. A simple algorithm for computing a set upper bound.

2.4. Example 1 continued

Consider the models from the motivating example. Let $\mathfrak{X}=\{1.00,1.01,\dots ,2.00\}$, ${\stackrel{\sim }{\theta }}_0=e$, and ${\stackrel{\sim }{\theta }}_1=1$. Note that these nominal values satisfy ${\eta }_0({\stackrel{\sim }{\theta }}_0,1)={\eta }_1({\stackrel{\sim }{\theta }}_1,1)$. Moreover, let us set ${\stackrel{\sim }{\mathrm{\Theta }}}_0^{(1)}=[e-1,e+1]$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1^{(1)}=[0,2]$, and let the required size of the experiment be $n=6$. First, we computed the value $o^2(\infty )\approx 0.02614$. Next, we used Algorithm 1 with $r_{ini}=0.3$ and $q=1+{10}^{-6}$, which returned a set upper bound $r^{\ast }\approx 0.6787$ after as few as seven computations of ${\delta }_r$‐optimal designs. Informed by $r^{\ast }$, we computed ${\delta }_r$‐optimal designs for $r=0.01,0.1,0.2,\dots ,0.7$. The resulting ${\delta }_r$‐optimal designs are displayed in Figure 3. Note that if ${\stackrel{\sim }{\mathrm{\Theta }}}^{(r)}$s are very narrow, the ${\delta }_r$‐optimal design is concentrated in the design point $x=2$, effectively maximizing the difference between ${\eta }_0({\stackrel{\sim }{\theta }}_0,x)$ and ${\eta }_1({\stackrel{\sim }{\theta }}_1,x)$. For larger values of r, the ${\delta }_r$‐optimal design has a 2‐point and ultimately a 3‐point support.

Figure 3.

${\delta }_r$‐Optimal designs of size $n=6$ for different rs; see the second part of the motivating example. The horizontal axis corresponds to the design space, and the vertical axis corresponds to different spans r of the flexible nominal sets. For each r, the figure displays the number of repeated observations at different design points, corresponding to the ${\delta }_r$‐optimal design

For some pairs of competing models there exists a set upper bound $r^{\ast }$, beyond which the values of ${\delta }_r$ are constantly 0 for all designs. These cases can be identified by solving a linear programming (LP) problem, as we show next.

Proposition 2.3

Let $\overline{\mathcal{D}}$ be the design that performs exactly one trial in each point of $\mathfrak{X}$. Consider the following LP problem with variables $r\in \mathbb{R}$, ${\theta }_0\in {\mathbb{R}}^m$, ${\theta }_1\in {\mathbb{R}}^m$:

$$\begin{array}{ccc}\hfill \min & & r\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& & {\mathbf{F}}_0(\overline{\mathcal{D}}){\theta }_0+{\mathbf{a}}_0(\overline{\mathcal{D}})={\mathbf{F}}_1(\overline{\mathcal{D}}){\theta }_1+{\mathbf{a}}_1(\overline{\mathcal{D}}),\hfill \\ & & {\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)},{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)},r\ge 0.\hfill \end{array}$$ (14)

Assume that (14) has some solution, and denote one solution of (14) by ${(r^{\ast },{\theta }_a^T,{\theta }_b^T)}^T$. Then, $r^{\ast }$ is a finite set upper bound. Moreover, $o(r)=0$ for all $r\in [r^{\ast },\infty ]$.

From the expression (7) we see that for any design $\mathcal{D}$ and its nonreplication version ${\mathcal{D}}^{nr}$ we have ${\delta }_r({\mathcal{D}}^{nr})=0$ implies ${\delta }_r(\mathcal{D})=0$. Moreover, if ${\mathcal{D}}_2⪰{\mathcal{D}}_1$ in the sense that ${\mathcal{D}}_2$ is an augmentation of ${\mathcal{D}}_1$ then ${\delta }_r({\mathcal{D}}_2)=0$ implies ${\delta }_r({\mathcal{D}}_1)=0$. Now let ${(r^{\ast },{\theta }_a^T,{\theta }_b^T)}^T$ be a solution of (14), let $r\ge r^{\ast }$ and let $\mathcal{D}$ be any design. Definition of ${\delta }_r$ and the form of (14) imply ${\delta }_r(\overline{\mathcal{D}})=0$. From $\overline{\mathcal{D}}⪰{\mathcal{D}}^{nr}$ we see that then ${\delta }_r({\mathcal{D}}^{nr})=0$, hence ${\delta }_r(\mathcal{D})=0$. The proposition follows.$\square$

Note that $r^{\ast }$ obtained using Proposition 2.3 does not depend on n, that is, it is a set upper bound simultaneously valid for all design sizes. The basic discriminability condition implies that $r^{\ast }\ne 0$.

If the competing models are linear, vectors ${\mathbf{a}}_0(\overline{\mathcal{D}})$ and ${\mathbf{a}}_1(\overline{\mathcal{D}})$ are zero. Therefore, (2.3) has a feasible solution ${(r,{\mathbf{0}}_m^T,{\mathbf{0}}_m^T)}^T$ for any $r\ge 0$ such that both ${\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)}$ and ${\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)}$ cover ${\mathbf{0}}_m$. That is, for the case of linear models, there is a finite set upper bound $r^{\ast }$ beyond which the ${\delta }_r$‐values of all designs vanish. However, the same holds for specific nonlinear models, including the ones from Section 3.

Proposition 2.4

Assume that both competing regression models are linear provided that we consider a proper subset of their parameters as known constants. Then (2.3) has a finite feasible solution, that is, there exists a finite set upper bound $r^{\ast }$ such that $o(r)=0$ for all $r\in [r^{\ast },\infty ]$.

Without loss of generality, assume that fixing the first $k_0<m$ components of θ₀ converts Model 0 to a linear model. More precisely, let ${\theta }_{01},\dots ,{\theta }_{0m}$ denote the components of θ₀ and assume that

$${\eta }_0({\theta }_0,x)=\sum _{j=k_0+1}^m{\gamma }_j^{(0)}({\theta }_{01},\dots ,{\theta }_{0k_0},x){\theta }_{0j}$$

for some functions ${\gamma }_j^{(0)}$, $j=k_0+1,\dots ,m$. Choose ${\widehat{\theta }}_0$ such that ${\widehat{\theta }}_{0j}={\stackrel{\sim }{\theta }}_{0j}$ for $j=1,\dots ,k_0$, and ${\widehat{\theta }}_{0j}=0$ for $j=k_0+1,\dots ,m$. Make an analogous assumption for Model 1 and also define ${\widehat{\theta }}_1$ analogously. It is then straightforward to verify that for the design $\overline{\mathcal{D}}$ from Proposition 2.3 we have ${\mathbf{F}}_k(\overline{\mathcal{D}}){\widehat{\theta }}_k+{\mathbf{a}}_k(\overline{\mathcal{D}})={\mathbf{0}}_d$, where $d=\#\mathfrak{X}$, for both $k=0,1$. Therefore, any ${(r,{\widehat{\theta }}_0^T,{\widehat{\theta }}_1^T)}^T$ such that ${\widehat{\theta }}_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)}$ and ${\widehat{\theta }}_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)}$ is a solution of (14) in Proposition 2.3.$\square$

In the following, we numerically demonstrate that the δ design criterion leads to designs which yield a high probability of correct discrimination.

3. AN APPLICATION IN ENZYME KINETICS

This real applied example is taken from Bogacka, Patan, Johnson, Youdim, and Atkinson (2011) and was already used in Atkinson (2012) to illustrate model‐discrimination designs. There two types of enzyme kinetic reactions are considered, where the reactions velocity y is alternatively modeled as

$$y=\frac{{\theta }_{01}{x}_1}{{\theta }_{02}\left(1+\frac{x_2}{{\theta }_{03}}\right)+x_1}+\epsilon$$ (15)

and

$$y=\frac{{\theta }_{11}{x}_1}{({\theta }_{12}+x_1)\left(1+\frac{x_2}{{\theta }_{13}}\right)}+\epsilon ,$$ (16)

which represent competitive and noncompetitive inhibition, respectively. Here x ₁ denotes the concentration of the substrate and x ₂ the concentration of an inhibitor. The data used in Bogacka et al. (2011) from an initial experiment of 120 observations are on Dextrometorphan–Sertraline and yields the estimates displayed in Table 1, where Gaussian errors were assumed. Assumed parameter spaces were not explicitly given there, but can be inferred from their figures as ${\theta }_{0,1},{\theta }_{1,1}\in (0,\infty )$, ${\theta }_{0,2},{\theta }_{1,2}\in (0,60]$, and ${\theta }_{0,3},{\theta }_{1,3}\in (0,30]$, respectively. Designs for parameter estimation in these models were recently given in Schorning, Dette, Kettelhake, and Möller (2017).

Table 1.

Parameter estimates and corresponding standard errors for models (15) and (16), respectively

	Estimate $\widehat{\theta }$	SE ${\widehat{\sigma }}_{\theta }$			Estimate $\widehat{\theta }$	SE ${\widehat{\sigma }}_{\theta }$
θ01	7.298	0.114		θ11	8.696	0.222
θ02	4.386	0.233		θ12	8.066	0.488
θ03	2.582	0.145		θ13	12.057	0.671

In Atkinson (2012), the two models are combined into an encompassing model:

$$y=\frac{{\theta }_{21}{x}_1}{{\theta }_{22}\left(1+\frac{x_2}{{\theta }_{23}}\right)+x_1\left(1+\frac{(1-\lambda )x_2}{{\theta }_{23}}\right)}+\epsilon ,$$ (17)

where $\lambda =1$ corresponds to (15) and $\lambda =0$ to (16), respectively. Following the ideas of Atkinson (1972) as used, for example, in Atkinson (2008) or Perrone, Rappold, and Müller (2017) one can then proceed to find so‐called $D_s$‐optimal (i.e., D‐optimal for only a subset of parameters) designs for λ and employ them for model discrimination. Note that also this method is not fully symmetric as it requires a nominal value for λ for linearization of (17), which induces some kind of weighting.

The nominal values used in Atkinson (2012) obviously motivated by the estimates of (15) were ${\stackrel{\sim }{\theta }}_{01}={\stackrel{\sim }{\theta }}_{11}={\stackrel{\sim }{\theta }}_{21}=10$, ${\stackrel{\sim }{\theta }}_{02}={\stackrel{\sim }{\theta }}_{12}={\stackrel{\sim }{\theta }}_{22}=4.36$, ${\stackrel{\sim }{\theta }}_{03}=2.58$, ${\stackrel{\sim }{\theta }}_{13}=5.16$, and ${\stackrel{\sim }{\theta }}_{23}=3.096$. However, note that particularly for model (16) the estimates in Table 1 give considerably different values and also nonlinear least squares directly on (17) yields the deviating estimates given in Table 2. The design region used was rectangular $\mathcal{X}={\mathcal{X}}_1\times {\mathcal{X}}_2=[0,30]\times [0,40]$.

Table 2.

Parameter estimates and corresponding standard errors for the encompassing model (17)

	Estimate $\widehat{\theta }$	SE ${\widehat{\sigma }}_{\theta }$
θ21	7.425	0.130
θ22	4.681	0.272
θ23	3.058	0.281
λ	0.964	0.019

In Table 2 of Atkinson (2012) four approximate optimal designs (we will denote them as A1–A4) were presented: the $T-$optimal designs assuming $\lambda =0$ (A1) and $\lambda =1$ (A4), a compound T‐optimal design (A3), and a $D_s$‐optimum (A2) for the encompassing model (for the latter note that Atkinson assumed $\lambda =0.8$, whereas the estimate suggest a much higher value). We will compare our δ‐optimal designs against exact versions of these designs, properly rounded by the method of Pukelsheim and Rieder (1992).

3.1. Confirmatory experiment $n=6$, normal errors

Let us first assume we want to complement the knowledge from our initial experiment by another experiment for which, however, we were given only limited resources, for example, for the sample sizes of only $n=6$ observations. Note that the aim is not to augment the previous 120 observations but to make a confirmatory decision just using the new observations. That is we are using the data from the initial experiment just to provide us with nominal values for parameter estimates and noise variances for the simulation, respectively. This is a realistic scenario if for instance for legal reasons the original data had to be deleted and only summary information was available.

As we are assuming equal variances for the two models we are using the estimate for the error standard deviation $\widehat{\sigma }=0.1526$ from the encompassing model as a base value for the simulation error standard deviation. However, using $\widehat{\sigma }$ was not very revealing for the discriminatory performance was consistently high for all designs. Thus, to accentuate the differences the actual standard deviation used was $2\times \widehat{\sigma }$ instead (unfortunately an even higher inflation is not feasible as it would result in frequent negative observations leading to faulty ML‐estimates). We then simulated the data‐generating process under each model for $N=10000$ times and calculated the total percentages of correct discrimination (hit rates) when using the likelihood ratio as decision rule.

We are comparing the designs A1–A4 to three specific δ designs $\delta 1,\delta 2$, and δ3, which represent a range of different nominal intervals. Specifically we chose ${\stackrel{\sim }{\mathrm{\Theta }}}_k=[{\stackrel{\sim }{\theta }}_{k1}\pm r{\stackrel{\sim }{\sigma }}_{k1}]\times [{\stackrel{\sim }{\theta }}_{k2}\pm r{\stackrel{\sim }{\sigma }}_{k2}]\times {[{\stackrel{\sim }{\theta }}_{k3}\pm r{\stackrel{\sim }{\sigma }}_{k3}]}_{k=0,1}$, where we chose ${\stackrel{\sim }{\theta }}_{kj}={\widehat{\theta }}_{kj}$ and ${\stackrel{\sim }{\sigma }}_{kj}={\widehat{\sigma }}_{kj}$ for $k=0,1$ and $j=1,2,3$. The tuning parameter r was set to three levels: $r=1$ (which is close to the lower bound of still providing a regular design), $r=5$ and $r=15$ (which is sufficiently close to the theoretical upper bound to yield a stable design), respectively. To make the latter more precise: the models in considerations are such that if we fix the last two out of the three parameters, then they become one‐parameter linear models. Therefore, using Proposition 2.4 we know that there exists a finite set upper bound $r^{\ast }$. Solving (14) provides the numerical value $r^{\ast }\approx 64.02$. Note that the same bound is valid for all design sizes n. Designs A1–A4 and δ1 all contain four support points, while δ2 has six and δ3 has five, respectively. A graphical depiction of the designs is given in Figure 4.

Figure 4.

Compared designs: first row A1–A4, second row δ1–δ3

Robustness study: As we would like to avoid comparing designs only when the data are generated from the nominal values (although this favors all designs equally), we perturbed the data‐generating process by drawing parameters from uniform distributions drawn at $\stackrel{\sim }{\theta }\pm c\times {\stackrel{\sim }{\sigma }}_{\theta }$, where c then acts as a perturbation parameter. Under these settings all these designs fare pretty well as can be seen from Table 3. However, $\mathrm{A}4$ and δ2 seem to outperform the other competing designs by usually narrow margins except perhaps for $\mathrm{A}1$, which is consistently doing worst. Note that in a real situation the true competitors of δ‐optimal designs are just $\mathrm{A}2$ and $\mathrm{A}3$ as it is unknown beforehand which model is true.

Table 3.

Total hit rates for $N=10000$ under each model, maximal values in boldface

c	0		1		5
True model	η0	η1	η0	η1	η0	η1
A1	91.11	94.45	91.35	93.95	90.44	93.24
A2	97.11	96.75	97.47	96.64	96.74	96.27
A3	96.60	96.51	96.47	96.40	95.69	96.06
A4	97.94	96.57	97.73	96.29	97.62	96.07
δ1	97.59	95.11	97.43	94.90	97.71	94.56
δ2	97.93	97.03	97.77	96.67	97.20	96.54
δ3	96.50	95.29	96.42	95.36	96.19	95.64

3.2. A second large‐scale experiment $n=60$, log‐normal errors

As the discriminatory power of all the designs for $n=60$ is nearly perfect, we are required to inflate the error variance. However, using additive normal errors in the data‐generating process and inflating the variance by a large enough factor, would generate a large number of negative observations, which renders likelihood estimation invalid. So, the data‐generating process was adapted to use multiplicative log‐normal errors. The observations were then rescaled to match the means from the original process. This way we are at liberty to inflate the error variance by any factor without producing faulty observations. Note that now the data‐generating process does not fully match the assumptions under which the designs were generated, but this can just be considered an extended robustness study as it holds for all compared designs equally. We could of course also have calculated the designs under the same data‐generating process, but as the fit of the model to the original data is not greatly improved and models (15) and (16) seem firmly established in the pharmacological literature, we refrained from doing this.

Perturbation of the parameters here did not exhibit a discernible effect, while the error inflation still does. For brevity, we here report only again the results for using $5\times \widehat{\sigma }$ (and $c=0$). The respective designs δ1–3 were qualitatively similar to those given in Figure 4 albeit with more diverse weights. In this simulation we generated 100 instances of $n=60$ observations from these designs a thousand times.

The corresponding boxplots of the correct classification rates are given in Figure 5. In this setting A4 seems a bit superior even under η₁ (remember it being the T‐optimum design assuming η₀ true), while δ1 and δ2 come close (and beat the true competitors A2 and A3) with A1 again being clearly the worst.

Figure 5.

Boxplot for the total correct classification rates for all designs using nominal values and error standard deviations of $5\times \widehat{\sigma }$; white under η₀, grey under η₁

4. CONCLUSIONS AND DIRECTIONS FOR FURTHER RESEARCH

We have presented a novel design criterion for symmetric model discrimination. Its main advantage is that design computations, unlike for T‐optimality, can be undertaken with efficient routines of quadratic optimization that in general enhance the speed of computations by an order of magnitude. An optimal exact design problem is a problem of discrete optimization, and the efficiency of its solution critically depends on the speed of evaluation of the design criterion. By a series of approximations, we substituted the theoretically ideal but numerically infeasible computation of the probability of correct discrimination with a simple convex optimization, which can be solved rapidly and reliably. Combined with the proposed methodology of flexible nominal sets, we can construct an entire sequence of exact experimental designs efficient for discrimination between models. Also it was shown in an example that resulting designs are competitive in their actual discriminatory abilities.

The notion of flexible nominal sets may have independent merit. Note again the distinction between parametric spaces and flexible nominal sets (and thus the principal distinction to “rigid” minimax approaches). Parametric spaces usually encompass all theoretically possible values of the parameters, while flexible nominal sets can contain the unknown parameters with very high likelihood, and still be significantly smaller than the original parameter spaces. In this paper, we do not completely specify the process of constructing the flexible nominal sets, but if we perform a two stage experiment, with a second, discriminatory phase, the potential specification through confidence intervals is an important problem.

As the approach suggested offers a fundamentally new way of constructing discriminatory designs, many properties are yet unexplored. A nonexhaustive list of questions follows.

Sequential procedure. The proposed method lends itself naturally to a two‐stage procedure, where parameter estimates and confidence intervals are employed as nominal values in the second stage. Even sequential generation of design points can be straightforwardly implemented.

Approximate designs. Proposition 2.1 is a possible gateway for the development of the standard approximate design theory for δ‐optimality because the criterion ${\delta }_{app}^2$ is concave on the set of all approximate designs. Therefore, it is possible to work out a minimax‐type equivalence theorem for δ‐optimal approximate designs, and use specific convex optimization methods to find a δ‐optimal approximate designs numerically. For instance, it would be possible to employ methods analogous to Burclová and Pázman (2016) or Yue, Vandenberghe, and Wong (2018).

Utilization of the δ‐optimal designs for related criteria. For a design $\mathcal{D}=(x_1,\dots ,x_n)$, a natural criterion closely related to ${\delta }_r$‐optimality can be defined as

$$\begin{array}{c}\hfill {\stackrel{\sim }{\delta }}_r(\mathcal{D})=\underset{{\theta }_0\in {\stackrel{\sim }{\mathrm{\Theta }}}_0^{(r)},{\theta }_1\in {\stackrel{\sim }{\mathrm{\Theta }}}_1^{(r)}}{inf}\stackrel{\sim }{\delta }(\mathcal{D}|{\theta }_0,{\theta }_1),\text{where}\\ \hfill \stackrel{\sim }{\delta }(\mathcal{D}|{\theta }_0,{\theta }_1)=‖{({\eta }_0({\theta }_0,x_i))}_{i=1}^n-{({\eta }_1({\theta }_1,x_i))}_{i=1}^n‖.\end{array}$$

The criterion ${\stackrel{\sim }{\delta }}_r$ requires a multivariate nonconvex optimization for the evaluation in each design $\mathcal{D}$, which entails possible numerical difficulties and a long time to compute an optimal design. However, the ${\delta }_r$‐optimal design, which can be computed rapidly and reliably, can serve as efficient initial design for the optimization of ${\stackrel{\sim }{\delta }}_r$. Note that if ${\stackrel{\sim }{\mathrm{\Theta }}}_0$ is a singleton containing only the nominal parameter value for Model 0, the ${\delta }_r$‐optimal designs could potentially be used as efficient initial designs for computing the exact version of the criterion of T‐optimality.

Selection of the best design from a finite set of possible candidates. As most proposals for the construction of optimal experimental designs, the method depends on the choice of some tuning parameters or even on entire prior distributions (in the Bayesian approach), which always results in a set of possible designs. It would be interesting to develop a comprehensive Monte‐Carlo methodology for the choice of the best design out of this pre‐selected small set of candidate designs. A useful generalization of the rule would take into account possibly unequal losses for the wrong classification.

Noncuboid sets. The methodology could certainly be extended to other types of flexible nominal sets, particularly when we are interested in functional relations among the parameters. However, then the particularly efficient box constrained quadratic programming algorithm could not be utilized.

Higher order approximations. As a referee remarked it is possible to employ tighter approximations of the sets of mean values of responses than the one that we suggest. For instance, it would be possible to use the local curvature of the mean‐value function. However, this may also lead to the loss of numerical efficiency of the method.

More than two rival models. Another referee remark leads us to point out the natural extension to investigate a weighted sum or the minimum δ over all paired comparisons. The implications of these suggestions, however, requires deeper investigations.

Different error variances. Yet another referee requested a clarification for how to proceed in case of unequal error variances for the two models. In case the functional form of these variances are known simple standardizations of the models will suffice. All other cases, including dependencies of the errors, will require more elaborate strategies.

Combination with other criteria. The proposed method can produce poor or even singular designs for estimating model parameters. Because of this problem, already mentioned in Atkinson and Fedorov (1975), Atkinson (2008) used a compound criterion called $DT$‐optimality. The same approach is possible for δ‐optimality. However, our numerical experience suggests that for a large enough size of the flexible nominal set, the δ‐optimal designs tend to be supported on a set that is large enough for estimability of the parameters, without any combination with an auxiliary criterion. A detailed analysis goes beyond the scope of this paper.

CONFLICT OF INTEREST

The authors declare that there is no conflict of interest.

Open Research Badges

This article has earned an Open Data badge for making publicly available the digitally‐shareable data necessary to reproduce the reported results. The data is available in the Supporting Information section.

This article has earned an open data badge “Reproducible Research” for making publicly available the code necessary to reproduce the reported results. The results reported in this article could fully be reproduced.

Supporting information

SUPPORTING INFORMATION

Harman R, Müller WG. A design criterion for symmetric model discrimination based on flexible nominal sets. Biometrical Journal. 2020;62:1090–1104. 10.1002/bimj.201900074