Optimality Criteria for the Design of 2-Color Microarray Studies

1 Introduction

Gene expression microarrays have become a standard technology in the biologist’s toolbox. For two-color microarrays, an immediate question in the early stages of planning a microarray study is how to choose the arrangement of samples onto the microarrays. Multiple papers have taken up this question, including Kerr and Churchill (2001); Dobbin and Simon (2002); Yang and Speed (2002); Kerr (2003a); Wit et al. (2005); Kerr (2006); Bailey (2007); Sanchez and Glonek (2009). Some of these papers have utilized the so-called ”alphabet” design criteria such as A- and D-optimality to evaluate microarray designs. Unfortunately, these design criteria are defined differently in the regression and block design settings, which seems to have caused some confusion in the literature.

This paper has two purposes. First, we explicate the definitions of A-, D-, and E-optimality to address the confusion in the literature. Second, we give our assessment of the relative value of these criteria for choosing microarray designs. From years of collaborating with biologists using microarrays, a guiding principle of our design recommendations will stem from our experience that a study that is most likely to be successful is one where particular comparisons are of interest a priori. It follows from this principle that design criteria should, as much as possible, focus on the comparisons of interest, which is a point of broad agreement.

2 Optimality Criteria Defined

Design optimality criteria are defined differently in the regression and block design settings. For each setting we provide the necessary background and then define the criteria.

2.1 Optimality Criteria in the Regression Setting

In linear regression, there is a model

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },$$

where y is the data vector, X is the design matrix, β is the (unknown) vector of regression parameters, and ε is the random error vector standardly assumed to have expectation 0 and cov(ε)=σ²I. In the regression setting it is reasonable to assume that X has full column rank; the model would not be worth fitting otherwise. This assumption implies that (X′X)⁻¹ exists, and that there is a unique least-squares estimate of β given by β̂ = (X′X)⁻¹X′y. Under the assumption of independent errors that have a common variance σ², β̂ has covariance σ²(X′X)⁻¹.

Some classical design optimality criteria are A-, D-, and E-optimality (Atkinson and Donev, 1992). The A-optimal design minimizes trace((X′X)⁻¹). This has the interpretation of minimizing the average variance among parameter estimates. The D-optimal design minimizes |(X′X)⁻¹|, or equivalently, maximizes |X′X|. This has the interpretation of minimizing the volume of the confidence ellipsoid for the regression parameters under the assumption that ε is multivariate normal. The E-optimal design minimizes the variance of the least-well estimated linear combination: a′β is minimized subject to a′a = 1. In the field of optimal design, ‘A’ stands for average, ‘D’ stands for determinant, and ‘E’ stands for extreme.

2.2 Optimality Criteria in the Block Design Setting

In block design there is also a linear model. Following the notation of John and Williams (1995), suppose there are υ treatments in b blocks of size k. In the microarray application, we consider each two-color microarray as the “block,” so k = 2. The different varieties of biological samples of RNA are the “treatments” and υ ≥ 2. For simplicity, suppose each treatment is replicated r times – some adjustment is needed when there are unequal replications. We write the model as

$$y_{ijk}=\mu +{\beta }_j+{\tau }_i+{\epsilon }_{ijk},$$

or, in matrix notation,

$$\mathbf{y}=\mathbf{1}\mu +\mathbf{Z}\mathit{\beta }+\mathbf{X}\mathit{\tau }+\mathit{\epsilon }.$$

μ is the model intercept, β is the vector of parameters β_j, 1 ≤ j ≤ b, representing block effects, and τ is the vector of parameters τ_i, 1 ≤ i ≤ υ, representing treatment effects. There are two important points. First, the overall design matrix

$$[\mathbf{1}|\mathbf{Z}|\mathbf{X}]$$

does not have full column rank and design criteria from general regression do not apply directly. Second, μ and β are nuisance parameters, so it is most appropriate to focus design questions on the treatment effects. More specifically, we translate the scientific question into a question about contrasts among treatment effects. Continuing to follow John and Williams (1995), the normal equations for the treatment effects are

$$\mathbf{A}\mathit{\tau ̂}=\mathbf{q},$$ (2.1)

where

$$\mathbf{A}=\text{diag}(r)-\mathbf{N}{(\text{diag}(k))}^{-1}\mathbf{N}\prime$$ (2.2)

and

$$\mathbf{q}=\mathbf{T}-\mathbf{N}{(\text{diag}(k))}^{-1}\mathbf{B}.$$ (2.3)

In (2.2) and (2.3), N is the incidence matrix of the design with υ rows and b columns: n_ij=1 if treatment i appears in block j and otherwise n_ij=0. T is the data vector of treatment totals and B is the data vector of block totals.

A is called the information matrix of the block design. The information matrix A always has a set of orthogonal eigenvectors p_i with corresponding eigenvalues θ_i. A is singular and exactly one eigenvalue is 0. We leave out the 0 eigenvalue and its eigenvector in forming

$$A^{+}=\sum _{i=1}^{\upsilon -1}{\theta }_i^{-1}{p}_i{p}_i^{\prime },$$

which is a generalized inverse of A. Since the eigenvectors are orthogonal and p_υ ∝ 1, this tells us that p₁, …, p_υ−1 are contrasts and therefore ${\mathbf{p}}_i^{\prime }\mathit{\tau }$ are estimable for i = 1, …, υ − 1. The ${\mathbf{p}}_i^{\prime }\mathit{\tau }$ are the basic contrasts of the design (Pearce et al., 1974). The variance of ${\mathbf{p}}_i^{\prime }\mathit{\tau ̂}$ is $\frac{{\sigma }^2}{{\theta }_i}$. Further, any contrast vector d can be written Σa_ip_i, and $\text{var}(\mathbf{d}\prime \mathit{\tau ̂})={\sigma }^2\sum \frac{a_i^2}{{\theta }_i}$.

Define the efficiency factor of a contrast as

$$\text{Eff}(\mathbf{d}\prime \mathit{\tau ̂})=\frac{{\text{var}}_{CBD}(\mathbf{d}\prime \mathit{\tau ̂})}{\text{var}(\mathbf{d}\prime \mathit{\tau ̂})},$$

where var_CBD(d′τ̂) is the variance of the contrast in an orthogonal design using the same number of experimental units¹. The efficiency factor of a basic contrast is $\text{Eff}({\mathbf{p}}_i^{\prime }\mathit{\tau ̂})=\frac{{\theta }_i}{r}\equiv e_i$. For a generic contrast we can write $\text{var}(\mathbf{d}\prime \mathit{\tau ̂})=\frac{{\sigma }^2}{r}\sum \frac{a_i^2}{e_i}$, and since e_i = 1 for a complete block design,

$$\text{Eff}(\mathbf{d}\prime \mathit{\tau ̂})=\frac{\frac{{\sigma }^2}{r}\sum \frac{a_i^2}{1}}{\frac{{\sigma }^2}{r}\sum \frac{a_i^2}{e_i}}=\frac{\sum a_i^2}{\sum \frac{a_i^2}{e_i}}.$$

Let θ_min be the smallest eigenvalue and θ_max be the largest. These correspond to the smallest and largest efficiency factors among the basic contrasts, e_min and e_max. Clearly

$$e_{\text{max}}^{-1}\sum a_i^2\le \sum \frac{a_i^2}{e_i}\le e_{\text{min}}^{-1}\sum a_i^2,$$

and thus

$$e_{\text{min}}\le \text{Eff}(\mathbf{d}\prime \mathit{\tau ̂})\le e_{\text{max}}.$$

Therefore, the canonical efficiency factors bound the efficiency of any contrast, and they can be considered a summary of the properties of a design. A design that maximizes the smallest canonical efficiency factor is said to be E-optimal. One can also choose a design to maximize the harmonic mean of the efficiency factors. Since it can be shown that the average variance over all pairwise treatment contrasts is proportional to ${\sum }_{i=1}^{\upsilon -1}\frac{1}{e_i}$ (Kempthorne, 1956), this design is called A-optimal. This is the classical definition of A-optimality for incomplete block designs – minimizing the average variance over pairwise treatment contrasts. This was the definition used by Kerr and Churchill (2001) to study microarray designs. A design that maximizes the geometric mean of the efficiency factors is D-optimal. The D-optimality criterion is the product of eigenvalues just like in the regression setting. However, in the regression setting we can interpret the D-criterion as a summary of the generalized variance of the regression parameters. We do not have the analogous interpretation in the block design setting – the D-criterion does not directly relate to the generalized variance of the model parameters (John and Williams, 1995, p. 32).

3 Regression-style D-Optimality Criteria for Microarrays

Linear models for the analysis of microarray data were first introduced as ANOVA models (Kerr et al., 2000; Wolfinger et al., 2001). Smyth (2004) and others introduced regression-style models. Briefly, by collapsing the data from a single microarray into logratios, an ANOVA-style model can be transformed into a regression-style model. The regression-style model uses parameters δ_ii′ ≡ τ_i − τ_i′, where the τ_i are the “treatment effects” in the ANOVA model. Kerr (2003b) describes in detail the relationship between the regression and ANOVA approaches in the analysis of microarray data. For analytic purposes, choice of a regression-style or ANOVA-style model is largely a matter of personal taste. However, the fact remains that there are different definitions of some alphabetic optimality criteria that are not equivalent.

Wit et al. (2005) employed regression-style models to evaluate microarray designs. They use υ − 1 pairwise treatment contrasts in a no-intercept model and consider A-, D-, and L-optimality as defined in the regression setting. It is important to note that Wit et al’s A- and D- criteria are not as traditionally defined in block design settings, generating some confusion. It is such confusion that this paper aims to ameliorate. The L-optimality criterion in Wit et al. (2005) is the criterion that this paper will advocate for.

Wit et al. (2005) note that D-optimality does not depend on the particular choice of treatment contrasts that parametrize the regression model. They imply that this invariance to parametrization is an advantage of D-optimality. We do not agree for three reasons. First, this invites investigators to naively seek a D-optimal design without considering the scientific question. As noted in the Introduction, we believe the scientific question should always be a driving force in the selection of a design. Second, the scientific question may not correspond to υ − 1 treatment contrasts, so invariance to choice of υ − 1 treatment contrasts is beside the point.

Our third reason for being less favorably inclined toward D-optimality is because its invariance leads to some indifference among designs. Consider a situation where there is a “control” treatment and three “experimental” treatments. Number the control treatment 1 and the experimental treatments 2, 3, 4. In the notation of Wit et al. (2005), scientific interest is in the parameters δ₁₂, δ₁₃, δ₁₄, i.e. the comparisons between the control treatment 1 and the experimental treatment 2, 3, and 4. This seems to be a case that is well-suited to the approach in Wit et al. (2005), since the contrasts of interest correspond to three parameters that can parametrize the regression model.

Consider the designs in Figure 1. Intuitively, Design A matches the scientific objective. In design A each of the experimental treatments is compared directly to the control treatment. Direct comparisons are most efficient, and it seems ideal to allot the three direct comparisons to the three pairwise contrasts that are of interest. Designs B and C are intuitively unappealing. However, D-Optimality says that all three designs are equally good. This can be viewed as a consequence of invariance to parametrization: if we parametrized the model instead with δ₁₄, δ₂₄, δ₃₄ then Design B becomes the more natural choice; if we parametrized the model with δ₁₂, δ₂₃, δ₃₄ then Design C becomes the natural choice. However, D-optimality is invariant to parametrization and so has no preference among these three designs.

Figure 1.

Designs for four varieties and three arrays/blocks. Designs are represented as undirected graphs. Each number represents a variety and each edge represents a microarray (a block of size 2).

In contrast to D-optimality and in accordance with intuition, if we consider the average variance among δ₁₂, δ₁₃, δ₁₄, Design A is strongly preferred. The average variance is 66% larger with Design B and twice as large with Design C. This is not to say that A-optimality is the ideal criterion, but in many block design problems we believe it is preferable to D-optimality.

As a second example, suppose there are two experimental treatments and one control treatment. The control treatment is included as way of checking that things are working as expected, but primary interest is in comparing the two experimental treatments. As before, label the control treatment 1 and the experimental treatments 2 and 3. Two possible designs are shown in Figure 2. Design A clearly matches the scientific objective of the study. The parameter of interest is δ₂₃; the variance of its estimate is 50% larger with Design B. However, D-optimality is indifferent between Designs A and B.

Figure 2.

Designs for three varieties and four arrays/blocks.

4 Block Design Optimality Criteria

There is broad agreement that a design criterion should match, as closely as possible, the scientific objectives of a study (Sanchez and Glonek, 2009; Wit et al., 2005). In a microarray study there are scientific questions of interest that correspond to contrasts of interest. In the previous section we argued against applying regression-style D-optimality. What about D- and E-optimality as defined in block design?

The E-optimal design maximizes the efficiency of the worst-case-scenario contrast. However, the worst-case-scenario contrast may have not relevance to the scientific question at hand. Similarly, the D-optimal design minimizes the generalized variance of the basic contrasts of the design (defined in Section 2.2). Again, the basic contrasts are not necessarily of scientific interest. In fact, the basic contrasts might be different from design to design. In total, we believe that block design D- and E-optimality fail to satisfy the basic requirement of reflecting the scientific objectives of a study.

5 Concluding Remarks

Definitions of the “alphabet” design criteria differ in the regression and block design settings, which has created some confusion in the literature. Wit et al. (2005) suggest that Kerr and Churchill (2001) use a non-standard definition of A-optimality. To the contrary, Kerr and Churchill (2001) use a definition consistent with the literature on block design. The approach in Wit et al. (2005) is unconventional in that it considers incomplete block designs in a regression setting. On the other hand, the approach in Sacan et al. (2009) is unclear since the design matrices in their formulation do not have full rank yet their stated design criteria are only defined for models with full-rank design matrices. Tan and Passos (2011) criticize the dependence of the A-optimality criterion on model parametrization but do not make it clear that their criticism applies to the definition of A-optimality in the regression setting, which is not the definition traditionally used in microarray design.

E- and D-optimal designs are advantageous in terms of the basic contrasts, but these are not the contrasts of interest. These criteria may be associated with good “all-purpose” designs. There are occasions when such designs may be appropriate. However, in our collaborations we prefer to choose designs that best-suit the scientific objectives of particular studies.

We have essentially argued for A-style optimality criteria when evaluating microarray designs. The A-optimality criteria can be applied to all pairwise contrasts, as in Kerr and Churchill (2001), or modified to apply to a different set of contrasts of interest. Wit et al. (2005), following Pukelsheim (1993), use the term L-optimality. A valid concern about A-optimality is that it does not account for the correlation among estimates. It is not clear how to address this, and we consider this an open question. However, for the reasons described, we do not consider D- optimality to be the solution.

Bailey (2007) points out that, unlike for larger block sizes, for block size two there can be severe conflict between the A- and D-criteria. She shows, for example, that the loop design is D-optimal for b = t ≥ 10, whereas the loop design is extremely A-inefficient. Similarly, the A-optimal design is severely D-inefficient. Bailey (2007, p. 376) proposes that robustness considerations can be used to choose between the A-optimal and D-optimal designs. We wholeheartedly endorse the consideration of robustness. However, given our general disfavor for D-optimality and preference for A-style optimality criteria directed at the contrasts of interest, we propose a slightly modified version of this strategy. We do not advocate automatic acceptance of the optimal design on the preferred criterion, but rather considering all efficient designs in light of robustness considerations. A slightly sub-optimal design might easily be preferable if it is more robust.