Small sample sizes: A big data problem in high-dimensional data analysis

Abstract

In many experiments and especially in translational and preclinical research, sample sizes are (very) small. In addition, data designs are often high dimensional, i.e. more dependent than independent replications of the trial are observed. The present paper discusses the applicability of max t-test-type statistics (multiple contrast tests) in high-dimensional designs (repeated measures or multivariate) with small sample sizes. A randomization-based approach is developed to approximate the distribution of the maximum statistic. Extensive simulation studies confirm that the new method is particularly suitable for analyzing data sets with small sample sizes. A real data set illustrates the application of the methods.

1 Introduction

Small sample sizes occur in various research experiments and especially in preclinical (animal) studies due to ethical, financial, and general feasibility reasons. Such studies are essential and an important part of translational medicine and other areas (e.g. rare diseases). Often, less than 20 animals per group are involved, and thus making valid inferences in these studies becomes a challenging part. In addition to the small sample sizes, repeated measurements as well as multiple endpoints are often observed on the experimental units (animals), naturally leading to a “large p, small n” situation and thus to a high-dimensional data design. Note that high-dimensional data do not only occur in animal studies, medical imaging and genomics are other well-known application areas. The first statistical problem at hand is neither the high dimensionality of the data nor the relatively low statistical power of the tests when sample sizes are very small—it is the accurate type-1 error rate control of the methods. Many of the existing statistical methods require moderate or large sample sizes and therefore tend to not control the type-1 error rate properly when sample sizes are very small; they either behave liberal and over-reject the null hypothesis or are conservative. Exact techniques (i.e. procedures that rely on the exact distribution of a test statistic for any finite sample size n) would be a great choice, but they typically rely on strict model assumptions that can hardly be verified—at least in more complex models. Indeed, making any assumptions about the underlying distributions (e.g. based on boxplots), verifying equality of variances, specific covariance structures, etc. is quasi impossible when sample sizes are so small and thus, methods, which do not rely on strict model assumptions, are the methods of choice. All in all, besides the often discussed problem of high dimensionality, small sample sizes increase the challenge of a robust and especially accurate data analysis in such situations.

Beyond these challenges and even though the statistical designs themselves are usually complex, the research questions and study aims are often very specific. These may be tackled by applying global testing procedures, which have been developed for different high-dimensional repeated measures and multivariate ANOVA models by several authors.^1–8 Furthermore, multivariate tests based on interpoint distances have been proposed.^9–12 Testing global null hypotheses and herewith answering the question whether any difference among the repeated measurements per or across endpoints exists, however, does usually not answer the main question of the practitioners—that is the specific localization of the responsible experimental conditions that lead to the overall significance conclusion. A modern data analysis requires the use of multiple comparison procedures that control the family-wise error rate in the strong sense and that are flexible in the way that they can be used to test arbitrary global and local null hypotheses and lead to compatible simultaneous confidence intervals (SCIs) for the underlying treatment effects. Furthermore, due to the often complex dependency structure across the repeated measurements and contrasts, the multiple comparison method should take the correlations of the different tests statistics for powerful data analysis into account. Such methods are also known as multiple contrast tests (MCTP) and are based on the maximum value of a vector of possibly correlated t-test type statistics (max t-test statistic). Computing its exact distribution without making strict distributional assumptions is impossible in general designs.¹³ Therefore, approximations of its asymptotic distribution are needed for making inferences. Recently, it has been suggested¹⁴ to estimate the distribution of the maximum value within a bootstrap simulation-based framework using the empirical correlation matrix. Simulation studies indicate, however, that sample sizes $n_i\ge 50$ are necessary for an accurate type-1 error rate control. When sample sizes are smaller, the methods tend to be liberal (see Section 5). In the present paper, a modification of the proposed method will be introduced that does not require the estimation of the correlation matrix. Extensive numerical studies show that the new approach controls the type-1 error very accurately even when sample sizes are very small and data do not follow multivariate normal distributions with equal covariance matrices.

The paper is organized as follows. In Section 2, a high-dimensional preclinical study on Azheimer’s disease with small sample sizes is described. Existing methodology for the statistical evaluation of such designs is explained in Section 3. Here, its behavior in small sample size situations is also investigated, which motivates the development of a different approximation of the distribution of the max t-test in Section 4. The qualities of the competing approximations are compared in extensive simulation studies in Section 5. The paper closes with the evaluation of the example and a discussion about the results in Sections 6 and 7, respectively. Theoretical properties of the new approximation and proofs are provided in the supplementary material file. Throughout the paper, ${\mathit{I}}_d$ denotes the d-dimensional unit matrix, ${\mathit{J}}_d=1_{d}1{\prime }_d$ the d × d matrix of 1 s, where $1_d=(1,\dots ,1){\prime }_{d\times 1}$.

2 A motivating example

As a motivating example, we consider a part of a preclinical study on Alzheimer's disease conducted in the Institute of Pharmacology at the Charité university medical center in Berlin, Germany. The study involves $n_1=10$ wild-type mice (group 1) and $n_2=9$ L1 tau-transgenic type (group 2) mice. As usual, the sample sizes of this preclinical research trial are pretty small. The abundance of each of the six different proteins Syntaxin, SNAP25, VAMP2, Synaptophysin, Synapsin-1 and Alpha-synuclein were measured in six different regions of the brain of every mouse. The regions of interest were pre-defined as hippocampal CA1 region (CA1), visual cortex (VC), medial septum (MS), vertical limb of the diagonal band of Broca (VDB), primary motor cortex (M1) and nucleus accumbens (ACB), respectively. Thus, 36 observations were made on every mouse, while the number of dependent replicates exceeds the number of independent replications of the trial. Therefore, the statistical design represents a classical “large p, small n” situation with small sample sizes. We note that generating such data requires very advanced technology. For graphical representation and easy display of the data, the protein abundances were log-transformed. The results are displayed in confidence interval plots (with chosen local level 95%) in Figure 1. For illustration, additional dotplots and boxplots of the data are displayed in the supplementary material file.

Figure 1.

Confidence interval plot (95%) for each protein× region× group combination in the protein abundance trial. Each confidence interval has been computed by inverting the corresponding one-sample t-test statistic using 97.5%-t-quantiles from a t-distribution with $n_i-1$ degrees of freedom.

The dotplots give the impression that the protein abundances are roughly symmetrically distributed. However, since sample sizes are so small, making any assumption about the underlying distribution would be questionable. It can also be seen that few “outliers” are present. These values have been kept in the data set, because the range of protein abundance measurements is usually very wide. Therefore, these values are not outliers in the classical sense and provide useful information about the protein levels in the respective brain regions. Furthermore, the confidence intervals displayed in Figure 1 show a fairly amount of variance heteroscedasticity. Therefore, the data of this trial can be modeled by independent and identically distributed random vectors

$${\mathit{X}}_{ik}=(X_{i1k},\dots ,X_{\mathit{idk}})\prime \sim {\mathit{F}}_i,i=1,2;k=1,\dots ,n_i;N=n_1+n_2$$ (1)

with expectation $E({\mathit{X}}_{i1})={\mathit{\mu }}_i=({\mu }_{i1},\dots ,{\mu }_{id})\prime$ and covariance matrix $\mathit{Cov}({\mathit{X}}_{\text{i1}})={\mathbf{\Sigma }}_{\text{i}}>\text{0}$, i = 1, 2. For a convenient notation, the index $s=1,\dots ,d$ represents the repeated measures in the regions under each of the different endpoints. Here, we set

$$d:\{\begin{array}{l}\text{Dimension}\text{Protein}\text{Region}\hfill \\ 1\le d\le 6\text{Syntaxin}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \\ 7\le d\le 12\text{SNAP25}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \\ 13\le d\le 18\text{VAMP2}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \\ 19\le d\le 24\text{Synaptophysin}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \\ 25\le d\le 30\text{Synapsin-1}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \\ 31\le d\le 36\text{Alpha-synuclein}\text{(CA1, VC, MS, VDB, M1, ACB)}\hfill \end{array}$$

and arrange all analyses according to this order. In general, data modeled by Equation (1) can either be repeated measures (measurements on the same scale), multivariate data (measurements on different scales) or combinations thereof. For a convenient notation of the hypotheses, let $\mathit{\mu }=({\mathit{\mu }}_1\prime ,{\mathit{\mu }}_2\prime )\prime$ denote the combined vector of the expectations in both groups. Besides the questions whether abundances of the proteins in the different regions of the brain differ, the study specifically aims to locate specific group × region interactions for each of the proteins. From a medical point of view, these would expose the proteins and especially the regions of the brain as biomedical biomarkers for Alzheimer’s disease. To be more specific, let ${\mu }_{ij}^{(P)}$ denote the expected protein abundance in group i under region j of protein P, where i = 1, 2; j $\in \{$CA1, VC, MS, VDB, M1, ACB$\}$ and $P\in \mathcal{P}=\{$Syntaxin, SNAP25, $\dots$ , Alpha-synuclein$\}$, respectively. For each single protein, the major aim is to (i) decide whether there is a group × region interaction and if so (ii) where. This can be achieved by simultaneously testing whether the group-wise differences ${\mu }_{1j}^{(P)}-{\mu }_{2j}^{(P)}$ are identical for all regions $j=1,\dots ,6$ for each protein $P=1,\dots ,6$. This leads to testing the family of 72 multiple null hypotheses

$$\mathbf{\Omega }=\left\{H_0^{(j,P)}:{\mu }_{ij}^{(P)}={\overline{\mu }}_{i·}^{(P)}+{\overline{\mu }}_{·j}^{(P)}-{\overline{\mu }}_{··}^{(P)},i=1,2,j=1,\dots ,6,P\in \mathcal{P}\right\}$$

at multiple level α. Here,

$${\overline{\mu }}_{i·}^{(P)}=\frac{1}{6}{\displaystyle \sum _{j=1}^6{\mu }_{ij}^{(P)}},{\overline{\mu }}_{·j}^{(P)}=\frac{1}{2}{\displaystyle \sum _{i=1}^2{\mu }_{ij}^{(P)}}\text{ and }{\overline{\mu }}_{··}^{(P)}=\frac{1}{12}{\displaystyle \sum _{i=1}^2{\displaystyle \sum _{j=1}^6{\mu }_{ij}^{(P)}}}$$

denote the corresponding means of expectations as known from linear model theory, where $i=1,2;j=1,\dots ,6;P\in \mathcal{P}$. Thus, the hypotheses are nothing but testing whether the differences of the expectations ${\mu }_{1j}^{(P)}-{\mu }_{2j}^{(P)},j=1,\dots ,6$ are identical for each protein. For simplicity, we rewrite the above using matrix notation and equivalently obtain

$$\mathbf{\Omega }=\{H_0^{(\ell )}:{\mathit{c}}_{\ell }\prime \mathit{\mu }=0,\ell =1,\dots ,q\}$$

where ${\mathit{c}}_{\ell }\prime$ denotes the $\ell$th row vector of the contrast matrix as used in

$$H_0^{\mu }:\mathit{C}\mathit{\mu }=0,\text{ with }\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{C}=\left(\underset{P\in \mathcal{P}}{{\displaystyle \oplus }}{\mathit{P}}_6\right)\left({\mathit{I}}_{36}⋮-{\mathit{I}}_{36}\right)$$ (2)

Here, in Equation (2), we denote with ${\mathit{P}}_m={\mathit{I}}_m-\frac{1}{m}{1}_m{1}_m\prime$ the m-dimensional centering matrix, while $\oplus$ describes the direct sum to build a block diagonal matrix.

Note that this summarizing matrix notation enables us to describe the 72 null hypotheses equivalently by only q = 36 which shall be tested using $n_1=10$ and $n_2=9$ independent replications. Therefore, the above is a high-dimensional multiple testing problem. An existing statistical method to analyze the data will be discussed in the next section.

3 Existing methodology

The high dimensionality of the testing problem considered here makes the data analysis complex in the sense that the computation of the critical values for making statistical inference becomes an issue. Recently, Chang et al.¹⁴ propose a simulation-based inference method for high-dimensional data. The procedure is valid for large dimensions and sample sizes. The case of small sample sizes has not been considered and therefore its applicability in such situations intrigues a detailed investigation. In their original paper, both the cases of studentized and non-studentized statistics have been considered. For the ease of read, we will concentrate on the studentized statistics in the following only. By doing so, we follow the guidelines of resampling studentized statistics.^15–18 First, we will rewrite the null hypothesis and introduce the statistics in the same way as they were described by Chang et al.¹⁴ who propose to test the equality of expectations of the q-variate random vectors ${\mathbf{Y}}_{ik}=(Y_{i1k},\dots ,Y_{\mathit{iqk}})\prime$ by $H_0^{\theta }:{\mathit{\theta }}_1={\mathit{\theta }}_2$. These considerations show that the two statistical testing problems are identical. However, we will propose a different way of estimating the critical values for making reliable inference later in Section 4.

Note that the null hypothesis $H_0:\mathit{C}\mathit{\mu }=0$ as given in Equation (2) can be equivalently written as the “standard” multivariate null hypothesis

$$H_0^{\theta }:{\mathit{\theta }}_1={\mathit{\theta }}_2$$

where ${\mathit{\theta }}_i=({\theta }_{i1},\dots ,{\theta }_{iq})\prime =E({\mathit{Y}}_{ik})$ denotes the expectation of the transformed vectors ${\mathit{Y}}_{ik}\prime ={\mathit{X}}_{ik}\prime \left({\oplus }_{s=1}^6{\mathit{P}}_6\right)$. In order to test $H_0^{\theta }$ against $H_1^{\theta }:{\mathit{\theta }}_1\ne {\mathit{\theta }}_2$, consider the maximum of the q component-wise t-test type statistics

$$T_0=\max \left\{\right|T_1|,\dots ,|T_q\left|\right\},\text{ where }{T}_{\ell }=\sqrt{N}\frac{({\overline{Y}}_{1\ell ·}-{\overline{Y}}_{2\ell ·})-({\theta }_{1\ell }-{\theta }_{2\ell })}{\sqrt{{\widehat{v}}_{1,\ell \ell }/n_1+{\widehat{v}}_{2,\ell \ell }/n_2}}$$ (3)

denotes the studentized difference of the means ${\overline{Y}}_{i\ell ·}=\frac{1}{n_i}{\displaystyle {\sum }_{k=1}^{n_i}{Y}_{i\ell k}}$ with the empirical variances ${\widehat{v}}_{i,\ell \ell }=N{(n_i-1)}^{-1}{\displaystyle {\sum }_{k=1}^{n_i}{(Y_{i\ell k}-{\overline{Y}}_{i\ell ·})}^2},i=1,2;\ell =1,\dots ,q$. The use of maximum t-statistics plays an important role in preclinical research, because local test decisions can be made using adjusted p-values for the comparisons $H_0^{(\ell )}:{\theta }_{1\ell }={\theta }_{2\ell },\ell =1,\dots ,q$. Second, each t-statistic describes the distance of the observed mean difference to its respective null hypothesis in units of standard deviations. However, for the computation of the local p-values, the distribution of T₀ must be known, at least approximately. Suppose for a moment that it is known, then the individual null hypothesis $H_0^{(\ell )}:{\theta }_{1\ell }={\theta }_{2\ell }$ will be rejected at multiple level α, if

$$\left|T_{\ell }\right|\ge z_{1-\alpha }(\max )$$ (4)

where $z_{1-\alpha }(\max )$ denotes the $(1-\alpha )$-quantile from the distribution of T₀. Compatible SCIs for the effects ${\delta }_{\ell }={\theta }_{1\ell }-{\theta }_{2\ell }$ are given by

$$C{I}_{\ell }=\left[{\overline{Y}}_{1\ell ·}-{\overline{Y}}_{2\ell ·}\mp \frac{z_{1-\alpha }(\max )}{\sqrt{N}}\sqrt{{\widehat{v}}_{1,\ell \ell }/n_1+{\widehat{v}}_{2,\ell \ell }/n_2}\right]$$ (5)

Finally, the global null hypothesis $H_0:\mathit{C}\mathit{\mu }=0$ will be rejected, if

$$T_0\ge z_{1-\alpha }(\max )$$ (6)

In such general models (even under the assumption of multivariate normality), however, the exact distribution of T₀ remains unknown¹⁹ and approximate methods are needed for estimating the distribution of T₀. In low-dimensional designs (fixed dimension d and contrasts q), the vector of t-statistics

$$\mathit{T}=(T_1,\dots ,T_q)\prime$$ (7)

follows, asymptotically, as $N\to \infty$, a multivariate normal distribution with expectation $0$ and correlation matrix

$$\mathit{R}={\mathit{D}}^{-1/2}\mathit{V}{\mathit{D}}^{-1/2},\text{ where }$$ (8)

$$\mathit{V}=\mathit{Cov}(\sqrt{\text{N}}({\overline{\mathit{Y}}}_{\text{1}·}\text{-}{\overline{\mathit{Y}}}_{\text{2}·}))\text{=N}\left(\ddot{\mathit{C}}{\mathbf{\Sigma }}_{\text{1}}\ddot{\mathit{C}}\prime {\text{/n}}_{\text{1}}\text{+}\ddot{\mathit{C}}{\mathbf{\Sigma }}_{\text{2}}\ddot{\mathit{C}}\prime {\text{/n}}_{\text{2}}\right)$$ (9)

with

$$\ddot{\mathit{C}}=\underset{s=1}{\overset{6}{\oplus }}{\mathit{P}}_6$$

denotes the covariance matrix of the differences in means and $\mathit{D}$ denotes the diagonal matrix obtained from the diagonal elements of $\mathit{V}$. These considerations show that the (asymptotic) joint distribution of the vector of t-statistics $\mathit{T}$ (and therefore of T₀) depends on the unknown correlation matrix and is non-pivotal. This is intuitively clear, since the higher the statistics are correlated, the smaller should be the critical value $z_{1-\alpha }(\max )$. Indeed, in case of a perfect correlation, the above reduces to a univariate testing problem. Anyway, the correlation matrix is unknown and the above cannot be used for making inferences in its present form. Chang et al.¹⁴ propose to first estimate the correlation matrix by its empirical counterpart

$$\widehat{\mathit{R}}={\widehat{\mathit{D}}}^{-1/2}\widehat{\mathit{V}}{\widehat{\mathit{D}}}^{-1/2},\text{ where }$$

$$\widehat{\mathit{V}}=N\left({\widehat{\mathit{V}}}_1/n_1+{\widehat{\mathit{V}}}_2/n_2\right),\text{ and }$$

$${\widehat{\mathit{V}}}_i=\frac{1}{n_i-1}{\displaystyle \sum _{k=1}^{n_i}({\mathit{Y}}_{ik}-{\overline{\mathit{Y}}}_{i·})({\mathit{Y}}_{ik}-{\overline{\mathit{Y}}}_{i·})}\prime ,i=1,2$$

denote the empirical covariance matrix of the random vectors ${\mathit{Y}}_{ik}$. Analogously, $\widehat{\mathit{D}}$ denotes the diagonal matrix obtained from the diagonal elements of $\widehat{\mathit{V}}$. Next, they propose to generate M random vectors

$${\mathit{Y}}_b^{\ast }\sim N(0,\widehat{\mathit{R}}),b=1,\dots ,M$$ (10)

from a multivariate normal distribution with expectation $0$ and correlation matrix $\widehat{\mathit{R}}$ and to estimate the critical value $z_{1-\alpha }(\max )$ by computing the $(1-\alpha )$-quantile $y_{1-\alpha }^{\ast }(\max )$ of the values $Y_{0,1}^{\ast },\dots ,Y_{0,M}^{\ast }$, where $Y_{0,b}^{\ast }=\max \left\{\right|Y_{b1}^{\ast }|,\dots ,|Y_{bq}^{\ast }\left|\right\}$ denotes the maximum value of each of the random vectors ${\mathit{Y}}_b^{\ast }$ (in absolute value). Finally, the unknown quantile $z_{1-\alpha }(\max )$ is replaced with the observable estimator $y_{1-\alpha }^{\ast }(\max )$ in Equations (4) to (6), respectively. Note that the quantile $y_{1-\alpha }^{\ast }$ can also be computed directly using the R-function qmvnorm implemented in the R-package mvtnorm²⁰ (if the dimension is not “too large”).

However, the present small sample sizes arise the question whether the method accurately controls the type-1 error rate and thus leads to reliable conclusions. Note that, in the data example, a 36 × 36 dimensional correlation matrix is estimated upon $n_1=10$ and $n_2=9$ independent vectors per group. Roughly speaking, the estimator might be too inaccurate when sample sizes are so small. In order to answer this question, a motivating simulation study has been conducted. Data has been generated from (i) multivariate normal and (ii) multivariate T₃-distributions with df = 3 degrees of freedom each with group specific covariance matrices ${\widehat{\mathit{V}}}_i$, i.e.

$$(i){\mathit{X}}_{ik}\sim N(\mathbf{0},{\widehat{\mathit{V}}}_i)\text{ and }(ii){\mathit{X}}_{ik}\sim T_3(\mathbf{0},{\widehat{\mathit{V}}}_i),i=1,2;k=1,\dots ,n_i$$ (11)

Here, ${\widehat{\mathit{V}}}_i$ denotes the empirical covariance matrix of group i in the Alzheimer’s disease study. Data have been transformed to ${\mathit{Y}}_{ik}\prime ={\mathit{X}}_{ik}\prime \left({\oplus }_{s=1}^6{\mathit{P}}_6\right)$ as described above. The T₃-distribution is heavy tailed and might be a reasonable candidate to mimic the distributional shape of the protein abundance data. Note that ${\widehat{\mathit{V}}}_i$ is singular and therefore data have been generated using singular value decomposition of ${\widehat{\mathit{V}}}_i$ using the rmvnorm function implemented in the mvtnorm R-package.²¹ The simulation results for varying sample sizes $n_1=n_2=8,9,\dots ,50$ at nominal significance level $\alpha =5\%$ are displayed in Table 1.

Table 1.

Type-1 error ($\alpha =5\%$) simulation results of the simulation-based test.¹⁴

Dist$\backslash n_i$	8	9	10	15	20	30	40	50
Nor	0.2134	0.1908	0.1723	0.1216	0.0975	0.0793	0.0720	0.0651
T 3	0.1681	0.1483	0.1133	0.0873	0.0781	0.0647	0.0430	0.0420

significance is provided with alpha = 5%

It follows that the procedure does not control the type-1 error rate appropriately when sample sizes are very small. With sample sizes n_i = 9, the empirical type-1 error rate is about 20% under normality and about 15% under heavy tailed $T_3(\mathbf{0},{\widehat{\mathit{V}}}_i)$-distribution and hence highly inflated. Only with larger sample sizes ($n_i\ge 50$), the method controls the type-1 error rate quite appropriately under normality, while it tends to be slightly conservative under $T_3(\mathbf{0},{\widehat{\mathit{V}}}_i)$, respectively. Digging for the reasons of this behavior, we first find that the procedure does not take the variations of the variance estimators used in the t-statistics in Equation (3) into account and second, the resampling algorithm is based on estimating the full correlation matrix of the vector of t-statistics. If a different resampling algorithm could be defined that overcomes both of these characteristics, a major improvement of the approximation might be available. In the next section, such a solution will be proposed.

4 Approximating the distribution of T₀

The arising challenge is finding a good approximation of the joint distribution of $\mathit{T}$ for estimating critical- and p-values. Resampling methods as above are an innovative way to do so. Roughly speaking, the corresponding test will work, if both the limiting and the resampling distributions of the statistic coincide—at least asymptotically—under the null hypothesis of no treatment effect. As explained above, the vector of t-test type statistics follows, asymptotically, a multivariate normal distribution with expectation $\mathbf{0}$ and correlation matrix $\mathit{R}$ in low-dimensional settings (d fixed). This means that a proper resampling algorithm must be designed in such a way that the resampling distribution of $\mathit{T}$, say ${\mathit{T}}^{\ast }$, converges to the $N(\mathbf{0},\mathit{R})$ distribution, respectively, where the correlation matrix must be identical to the one defined in Equation (8). Moreover, in high-dimensional settings (with $d\to \infty$) similar observations apply, see the supplementary material, where it is, for example, shown that the distribution of $\mathit{T}$ converges to a discrete Gaussian process. Detailed assumptions, especially on the covariance matrices, are listed in the supplementary material document as well. Having these thoughts in mind, not every resampling method is applicable in high-dimensional designs with an emphasis on small sample sizes. For example, the nonparametric bootstrap (drawing with replacement) shows poor finite sample performances in a similar setting under stronger conditions.²² Moreover, the therein proposed permutation method for exchangeable designs is in general not applicable in our unbalanced heteroscedastic setting.^23,24 For more details on permutation tests, we refer to existing overviews and monographs.^25–28 Therefore, the generation of the resampling variables and especially the algorithmic build-up play an important role for achieving a adequate bootstrap test in high-dimensional designs. Furthermore, even in low-dimensional settings (d fixed), estimating the approximate null $N(0,\mathit{R})$ distribution of $\mathit{T}$ using a plug-in estimator $\widehat{\mathit{R}}$ usually requires large sample sizes for an appropriate approximation. We therefore propose to approximate the limiting distribution of $\mathit{T}$ without estimating the parameter of the distribution using a Wild-bootstrap randomization approach that is applicable in low- as well as high-dimensional situations. The method follows the same ideas proposed for matched pairs^18,29 and in high-dimensional linear models,³⁰ and is as follows. Let

$${\mathit{Z}}_{ik}={\mathit{Y}}_{ik}-{\overline{\mathit{Y}}}_{i·},i=1,2;k=1,\dots ,n_i$$

denote the centered random vectors and let W_ik denote N independent and identically distributed random signs with $P(W_{ik}=\pm 1)=\frac{1}{2}$. Now, let

$${\mathit{Z}}_{ik}^{*}=W_{ik}{\mathit{Z}}_{ik}$$

denote the resampling variables, ${\overline{\mathit{Z}}}_{i·}^{\ast }=\frac{1}{n_i}{\displaystyle {\sum }_{k=1}^{n_i}{\mathit{Z}}_{ik}^{\ast }}=({\overline{Z}}_{i1·}^{\ast },\dots ,{\overline{Z}}_{iq·}^{\ast })\prime$ their empirical means and let

$${\widehat{v}}_{i,\ell \ell }^{\ast }=N\frac{1}{n_i-1}{\displaystyle \sum _{k=1}^{n_i}{(Z_{i\ell k}^{\ast }-{\overline{Z}}_{i\ell ·}^{\ast })}^2},i=1,2\text{ and }\ell =1,\dots ,q$$ (12)

denote the empirical variance of the variables obtained under the $\ell$th condition. Now, the resampling version of the original test statistic T₀ is given by

$$T_0^{\ast }=\max \left\{\right|T_1^{\ast }|,\dots ,|T_q^{\ast }\left|\right\},\text{ where }{T}_{\ell }^{\ast }=\sqrt{N}\frac{({\overline{Z}}_{1\ell ·}^{\ast }-{\overline{Z}}_{2\ell ·}^{\ast })}{\sqrt{{\widehat{v}}_{1,\ell \ell }^{\ast }/n_1+{\widehat{v}}_{2,\ell \ell }^{\ast }/n_2}}$$ (13)

In comparison to the existing methodology discussed in Section 3, the statistic $T_0^{\ast }$ mimics the computational process that lead to the original statistic T₀. Moreover, it is shown in the supplementary material that for both low- and high-dimensional settings, the conditional distribution of the vector of statistics (given the data $\mathit{X}$)

$${\mathit{T}}^{\ast }=(T_1^{\ast },\dots ,T_q^{\ast })\prime$$ (14)

mimics the null distribution of $\mathit{T}$. For making statistical inference, the critical value $z_{1-\alpha }(\max )$ is now estimated by the following steps:

1. Fix the data $\mathit{X}$ (or $\mathit{Y}$) and compute the centered variables ${\mathit{Z}}_{ik}$.

2. Generate random weights W_ik, compute the resampling variables ${\mathit{Z}}_{ik}^{\ast }$, the test statistics ${\mathit{T}}^{\ast }$ and safe the value of $T_0^{\ast }$ in $T_{0,b}^{\ast }$.

3. Repeat the previous step a large number of times (e.g. M = 10, 000) and compute the values $T_{0,1}^{\ast },\dots ,T_{0,M}^{\ast }$.

4. Estimate $z_{1-\alpha }(\max )$ by the empirical $(1-\alpha )$-quantile $z_{1-\alpha }^{\ast }(\max )$ of $T_{0,1}^{\ast },\dots ,T_{0,M}^{\ast }$.

Finally, the unknown quantile $z_{1-\alpha }(\max )$ is replaced with the observable value $z_{1-\alpha }^{\ast }(\max )$ in Equations (4) to (6), respectively. One-sided tests and p-values are estimated analogously. The estimation of $z_{1-\alpha }(\mathit{R})$ thus gets by without estimating the full correlation matrix $\mathit{R}$ and additionally takes the variability of the variance estimators into account. Note that the set $\{H_0^{(\ell )},T_{\ell },\ell =1,\dots ,q\}$ consisting of the null hypotheses and corresponding test statistics constitutes a joint testing family in the sense of Gabriel.³¹ Therefore, the simultaneous test procedure controls the family-wise error rate in the strong sense asymptotically in case of fixed q. Its accuracy in terms of controlling the type-1 error rate and power to detect alternatives when sample sizes are small will be investigated in the next section.

Remark: Throughout the manuscript, we consider the maximum statistic T₀ as a combination of the q possibly correlated test statistics only. It originates from finding appropriate real valued $c_{1-\alpha }$ such that

$$P\left({\displaystyle \underset{\ell =1}{\overset{q}{\cap }}\left\{-c_{1-\alpha }\le T_{\ell }\le c_{1-\alpha }\right\}}\right)=1-\alpha \iff P\left({\max }_{\ell =1}^q\left|T_{\ell }\right|\le c_{1-\alpha }\right)=1-\alpha$$

The right-hand side holds with $z_{1-\alpha }(\max )=c_{1-\alpha }$. The resulting test further allows inversion of the test statistics into simultaneous confidence intervals. However, we note that other combining functions than computing the maximum statistic would be possible, for instance Fisher’s weighted combining function. A general overview of nonparametric combination terminologies are provided in the monographs of Pesarin and Salmaso³² and Salmaso et al.²⁷ and the references therein.

5 Simulations

In this section, we investigate the small sample properties of the proposed randomization technique within extensive simulation studies. The study aims to compare the two different approximations of the distribution of T₀ presented in the paper. As the true distribution of T₀ remains unknown, the type-1 error control of the competing methods will be used as a quality criterion. Later, the all-pairs and the any-pairs powers of the two methods will be compared. We conducted the extensive simulation studies in R (version 3.6.1). Marozzi¹² discusses different methods to compute the numbers nsim and nboot of simulation and resampling runs in detail. Using his result and under some assumptions, nsim = 10,000 simulations lead to $\mathit{nboot}=8\sqrt{\mathit{nsim}}=800$ resampling runs and a maximal simulation error of $0.006\approx 1\%$. Since the methods proposed in the manuscript are computationally feasible, we chose nsim = 10,000 and nboot = 1000 runs for each setting. Furthermore, setting $\alpha =5\%$, we can compute the 95% precision interval $[5\%\mp \frac{1.96}{\sqrt{1,000}}\sqrt{5\%*95\%}]=[4.6\%;5.4\%]$. If the empirical type-1 error of a test is within this interval, the method can be seen as accurate. The simulation code is displayed in the supplementary material file for reproducibility.

5.1 Type-1 error simulation results

Due to the abundance of possible factorial designs and hypotheses, two-way designs with varying dimension $d\in \{2,4,\dots ,150\}$ will be simulated and the hypothesis $H_0:{\mathit{P}}_d({\mathit{\mu }}_1-{\mathit{\mu }}_2)=0$ of no interaction effect will be tested at 5% level of significance. Data were generated from model

$${\mathit{X}}_{ik}\sim {\mathit{F}}_i({\mathit{\mu }}_0,{\mathbf{\Sigma }}_i)+{\mathit{\mu }}_i,i=1,2,k=1,\dots ,n_i,$$ (15)

where ${\mathit{F}}_i({\mathit{\mu }}_0,{\mathbf{\Sigma }}_i)$ represents a multivariate distribution with expectation vector ${\mathit{\mu }}_0$, correlation matrix ${\mathbf{\Sigma }}_i$ and location shifts ${\mathit{\mu }}_i$. As representative marginal data distributions, we selected three differently tailed symmetric distributions (normal, logistic, T₃) and three skewed distributions (ranging from mildly to very skewed) (${\chi }_7^2,{\chi }_{15}^2$, exponential) each with sample sizes $n_i\in \{10,20\}$. A major assessment criteria of the quality of the proposed approximations is the impact of both the chosen contrast as well as the dependency structures of the data—especially when data has different covariance matrices and thus covering a typical Behrens-Fisher situation. Here, we used normal copulas in order to generate rather complex dependency structures of the repeated measurements using the R-package copula.³³ The different allocations of the correlation matrices used in the simulation studies are summarized in Table 2.

Table 2.

Different correlation matrices used in the simulation study.

Setting 1:	${\mathbf{\Sigma }}_1=({\sigma }_{1,ij})={0.6}^{|i-j|}$	${\mathbf{\Sigma }}_2=({\sigma }_{2,ij})={0.6}^{|i-j|}$
Setting 2:	${\mathbf{\Sigma }}_1=({\sigma }_{1,ij})={0.6}^{|i-j|/(d-1)}$	${\mathbf{\Sigma }}_2=({\sigma }_{2,ij})={0.6}^{|i-j|}$
Setting 3:	${\mathbf{\Sigma }}_1=({\sigma }_{1,ij})=1-|i-j|/d$	${\mathbf{\Sigma }}_2=({\sigma }_{2,ij})={0.6}^{|i-j|/(d-1)}$
Setting 4:	${\mathbf{\Sigma }}_1={\mathit{I}}_d+0.5·({\mathit{J}}_d-{\mathit{I}}_d)$	${\mathbf{\Sigma }}_2={\mathit{I}}_d+0.25·({\mathit{J}}_d-{\mathit{I}}_d)$.

In Setting 1, both correlation matrices ${\mathbf{\Sigma }}_1$ and ${\mathbf{\Sigma }}_2$ are identical and represent an autoregressive structure. In Settings 2 and 3, the covariance matrices ${\mathbf{\Sigma }}_1$ and ${\mathbf{\Sigma }}_2$ have different off-diagonal elements models, whereas an autoregressive structure depending on the dimension d is modeled by ${\mathbf{\Sigma }}_1$ in Setting 2, and a linearly decreasing (symmetric) Toeplitz structure is covered by ${\mathbf{\Sigma }}_1$ in Setting 3 (see Table 2), see Pauly et al.⁷ for similar choices. Note that Setting 2 models a pretty extreme scenario. For a detailed overview of copulas, we refer to Nelsen³⁴ or Marozzi.³⁵

All these four settings will be simulated for all four sample sizes ($n_i\in \{10,20\}$), dimensions ($d\in \{2,4,\dots ,150\}$) and distributional configurations as described above. The type-1 error simulation results obtained under Setting 1 are displayed in Figure 2. All others are displayed in the supplementary material file.

Figure 2.

Type-1 error ($\alpha =5\%$) simulation results of the Wild-bootstrap randomization test $\mathit{T}$ in Equation (14) (Wild) and simulation-based test $\mathit{T}$ in Equation (10) (Chang). Data have covariance matrices as described in Setting 1 in Table 2.

First, it can be seen that the underlying covariance matrices significantly impact the accuracy of the simulation-based procedure proposed by Chang et al.¹⁴ in small sample size situations. It can also readily be seen that this test shows an increasing liberal behavior for increasing dimension d. The over-rejection of the hypotheses occurs, because the test decision is based upon quantiles from the $N(0,\widehat{\mathit{R}})$ distribution, which neither takes the variability nor the distribution of the variance estimators into account. On the contrary, the randomization-based test T₀ in Equation (13) tends to control the nominal type-1 error rate very well, even in case of very small sample sizes and large dimensions. The underlying covariance structures seem to impact the results only minor (if even). In case of mildly skewed distributions, the simulation results indicate that the resampling test controls the type-1 error accurately. However, in case of skewed data with different covariance matrices, the test might be very liberal when sample sizes are small. This behavior especially depends on the type of contrast of interest and whether it induces positive or negative correlations. The simulation results obtained under Setting 2 (see the supplementary material) indicate that none of the methods should be applied in these (rather extreme) cases in practice. Other methods, e.g. nonparametric methods rather than mean-based procedures, might be more appropriate for the analysis of small skewed data in such cases. However, the liberality vanishes with increasing sample sizes.³⁶ In the three other settings considered here, the randomization procedure controls the type-1 error rate accurately, even when sample sizes are small and data follow skewed distributions. Pauly et al.²⁴ report similar conclusions for general linear models with independent observations. Furthermore, in all of the settings considered here, the randomization-based resampling method is accurate when data is heavy-tailed but symmetric. As many factors might impact the behavior of the tests, however, the procedure might be sensitive to such distributional shapes in different scenarios than the ones considered here. For the generation of other copula models, e.g. Elliptical and Archimedean copulas, we refer to Marozzi.³⁵

Remark: Instead of using copulas for generation of the multivariate distributions, an alternative method is generating data from model

$${\mathit{X}}_{ik}={\mathit{\mu }}_i+{\mathbf{\Sigma }}_i^{-1/2}{\mathit{ϵ}}_{ik},i=1,2,k=1,\dots ,n_i$$

where the error terms is generated from standardized distributions, respectively. Additional s''imulation studies indicate that the empirical behavior of the test procedures is very similar.

5.2 Power simulation results

Next the all-pairs power P (“reject all false null hypotheses”) as well as the any-pairs powers P (“reject any true or false null hypothesis”) of the competing methods to reject the null hypothesis $H_0:{\mathit{P}}_d({\mathit{\mu }}_1-{\mathit{\mu }}_2)=0$ ($\alpha =5\%$) will be simulated for selected alternatives. The aim of the simulation study is investigating the impact of the underlying distributions, dependency structures of the data, sample size allocations and dimensions on the powers of the tests. Data have been generated (under the alternative) from model Equation (15) with expectations

$${\mathit{\mu }}_1=(\delta ,2\delta ,\delta ,0,\dots ,0)\prime \text{ and }{\mathit{\mu }}_2=(2\delta ,\delta ,2\delta ,0,\dots ,0)\prime$$

and varying $\delta \in \{0,0.1,\dots ,2\}$ from the same six distributions as above (normal, logistic, T₃, ${\chi }_7^2,{\chi }_{15}^2$ and exponential) having all of the four different dependency structures displayed in Table 2, respectively. The dimension of the random vectors was set to d = 30. Due to the liberality of Chang et al.’s method for small sample sizes, large sample sizes were simulated (n_i = 100) in order to be able to compare the powers of the methods on a fair basis, i.e. in a situation where both of them control the type-1 error rate accurately. For illustration, an additional power simulation with small sample sizes (n_i = 10) has been conducted. First, it turns out that the types of covariance structures affect the powers of the tests. This is not surprising, because the higher the correlation the smaller are the variances of the effect size estimators. Overall, the simulation results indicate that the competing methods have comparable powers when sample sizes are large. Chang et al.’s method has slightly larger any-pairs and all-pairs powers than the randomization test (about 1% higher). However, when sample sizes are small, the randomization method controls the size and has a reasonable power. The simulations of the all-pairs power furthermore indicate the strong control of the FWER of both methods. Under the situations considered here, the shapes of the underlying distributions impact the results. As expected, the power of the methods under ${\chi }^2$-distributions appears to be rather low. The reason is the pretty large variance of the ${\chi }^2$-distribution compared with the other distributions. It should be noted that the above findings only hold for the settings considered here and might be different under other scenarios. The all-pairs and the any-pairs power curves are displayed in the supplementary material file.

6 Evaluation

The extensive simulation studies show that the newly proposed randomization test controls the type-1 error rate very satisfactorily, even when sample sizes are very small and data do not follow a multivariate normal distribution. In a first step, we perform a further type-1 error simulation study and investigate the accuracy of the method for analyzing this specific data set in the same way as presented in Table 1. As before in Equation (11), we mimic the data set using multivariate normal $N(\mathbf{0},{\widehat{\mathit{V}}}_i)$ and multivariate $T_3(\mathbf{0},{\widehat{\mathit{V}}}_i)$ distributions, respectively. The type-1 error simulation results are displayed in Table 3. It appears that the method controls the type-1 error accurately. The high-dimensional preclinical study on Alzheimer’s disease introduced in Section 2 can therefore now be analyzed with this method. For comparisons, the data set will be analyzed using both of the discussed approximations.

Table 3.

Type-1 error ($\alpha =5\%$) simulation results of the Wild-bootstrap randomization test.

Dist$\backslash n_i$	8	9	10	15	20	30	40	50
Nor	0.0530	0.0512	0.0494	0.0496	0.0474	0.0502	0.0501	0.0497
T 3	0.0392	0.0411	0.0431	0.0464	0.0458	0.0512	0.0527	0.0502

significance is provided with alpha = 5%

In addition to testing for interactions motivated in Equation (2), multiple comparisons inferring the region as well as the group effects are of interest. These will be performed using the contrast matrices

$$H_0^{\mu }:\mathit{C}\mathit{\mu }=0,\text{ where }\mathit{C}=\left({\oplus }_{s=1}^6{\mathit{P}}_6\right)\left({\mathit{I}}_{36}⋮{\mathit{I}}_{36}\right)\text{(Region)},\text{ and }$$

$$H_0^{\mu }:\mathit{C}\mathit{\mu }=0,\text{ where }\mathit{C}=\left({\mathit{I}}_{36}⋮-{\mathit{I}}_{36}\right)(\text{Group})$$ (16)

Note that for testing the impact of the region, the test statistics are given by

$$T_{\ell }=\sqrt{N}\frac{{\overline{Y}}_{1\ell ·}+{\overline{Y}}_{2\ell ·}-({\theta }_{1\ell }+{\theta }_{2\ell })}{\sqrt{{\widehat{v}}_{1,\ell \ell }/n_1+{\widehat{v}}_{2,\ell \ell }/n_2}}$$

where ${\overline{Y}}_{i\ell ·}$ denotes the mean of the $\ell$th component of the vector ${\mathit{Y}}_{ik}\prime ={\mathit{X}}_{ik}\prime \left({\oplus }_{s=1}^6{\mathit{P}}_6\right)$. Therefore, the correlation matrix of the vector of test statistics $\mathit{T}=(T_1,\dots ,T_q)\prime$ is identical to the one using interaction contrasts as described in Section 3. The randomization approach for approximating the joint null distribution of these statistics is adapted accordingly. Testing for the group effects is the “standard” multivariate hypothesis. Means and empirical variances of the protein abundance data under each protein × region × group combination are provided in the supplementary material file.

As already indicated by the confidence interval plots in Figure 1, data show a fairly amount of variance heteroscedasticity. Therefore, assuming equal covariance matrices across the groups is doubtful. Next, the multiple hypotheses will be tested using the two different approaches. The point estimators of the contrasts in means ${\widehat{\delta }}_{\ell }$, values of the test statistics $T_{\ell }$ equation (3), the estimated quantile $z_{95\%}(\max )$ as well as 95%-simultaneous confidence intervals equation (5) using both the simulation as well as randomization technique will be displayed for all of the three multiple hypotheses. In total, M = 100,000 simulation and randomization runs have been performed. The results are displayed in Table 4. Different decisions (at 5% level) between the two competing methods are highlighted in boldface.

Table 4.

Multiple contrast test results of the protein abundance trial.

	Interaction						Region						Group
	Chang ($z_{95\%}=3.10)$			Randomization ($z_{95\%}=3.76$)			Chang ($z_{95\%}=3.10)$			Rand. ($z_{95\%}=3.75$)			Chang ($z_{95\%}=3.09)$			Rand. $(z_{95\%}=3.88)$
$\ell =$	${\widehat{\delta }}_{\ell }$	$T_{\ell }$	95%-L	95%-U	95%-L	95%-U	${\widehat{\delta }}_{\ell }$	$T_{\ell }$	95%-L	95%-U	95%-L	95%-U	${\widehat{\delta }}_{\ell }$	$T_{\ell }$	95%-L	95%-U	95%-L	95%-U
1	−0.65	−1.20	−2.32	1.03	−2.67	1.37	1.35	2.50	−0.32	3.03	−0.68	3.38	0.39	0.41	−2.57	3.36	−3.32	4.11
2	−0.53	−1.00	−2.17	1.11	−2.50	1.44	2.49	4.71	0.85	4.13	0.51	4.48	0.51	0.69	−1.77	2.80	−2.35	3.38
3	1.09	1.37	−1.37	3.55	−1.88	4.05	−2.38	−3.00	−4.84	0.07	−5.36	0.59	2.13	1.93	−1.28	5.55	−2.15	6.42
4	−0.04	−0.10	−1.41	1.32	−1.69	1.60	−2.40	−5.43	−3.77	−1.03	−4.06	−0.74	1.00	1.06	−1.92	3.91	−2.66	4.66
5	−0.01	−0.02	−1.39	1.37	−1.67	1.66	−0.68	−1.53	−2.06	0.70	−2.36	0.99	1.04	1.03	−2.08	4.15	−2.87	4.94
6	0.14	0.28	−1.41	1.70	−1.74	2.02	1.62	3.23	0.06	3.18	−0.26	3.51	1.18	1.20	−1.85	4.21	−2.62	4.99
7	−0.22	−0.72	−1.15	0.71	−1.34	0.91	0.18	0.58	−0.76	1.11	−0.95	1.30	−0.61	−1.84	−1.64	0.41	−1.90	0.68
8	−0.68	−2.50	−1.53	0.16	−1.71	0.34	−1.85	−6.75	−2.70	−1.00	−2.88	−0.82	−1.08	−2.90	−2.23	0.07	−2.53	0.36
9	0.15	0.74	−0.49	0.80	−0.63	0.94	−0.10	−0.50	−0.75	0.54	−0.89	0.68	−0.24	−0.89	−1.09	0.60	−1.30	0.82
10	0.20	0.98	−0.43	0.83	−0.56	0.96	−1.97	−9.69	−2.60	−1.34	−2.74	−1.21	−0.20	−0.85	−0.91	0.52	−1.10	0.70
11	0.24	1.41	−0.28	0.75	−0.39	0.86	1.10	6.58	0.58	1.62	0.47	1.73	−0.16	−0.61	−0.98	0.66	−1.19	0.87
12	0.31	2.42	−0.09	0.71	−0.17	0.79	2.65	20.58	2.25	3.05	2.17	3.13	−0.09	−0.77	−0.43	0.25	−0.51	0.34
13	0.46	1.44	−0.53	1.45	−0.74	1.66	2.62	8.18	1.63	3.62	1.42	3.83	0.59	1.66	−0.51	1.69	−0.79	1.97
14	−0.07	−0.31	−0.79	0.64	−0.93	0.79	3.73	16.17	3.02	4.45	2.87	4.60	0.06	0.24	−0.70	0.82	−0.90	1.02
15	−0.23	−0.29	−2.67	2.21	−3.18	2.72	−6.67	−8.45	−9.12	−4.23	−9.63	−3.71	−0.10	−0.09	−3.37	3.17	−4.20	4.00
16	−0.41	−1.80	−1.13	0.30	−1.27	0.45	−3.98	−17.27	−4.69	−3.26	−4.84	−3.11	−0.28	−0.62	−1.70	1.14	−2.07	1.50
17	0.24	0.55	−1.11	1.59	−1.39	1.87	1.46	3.34	0.11	2.81	−0.18	3.10	0.37	0.98	−0.80	1.54	−1.10	1.84
18	0.01	0.04	−0.91	0.94	−1.10	1.13	2.83	9.47	1.90	3.75	1.71	3.95	0.14	0.45	−0.84	1.13	−1.09	1.38
19	−0.57	−1.11	−2.14	1.01	−2.47	1.34	−1.78	−3.49	−3.36	−0.20	−3.69	0.13	−0.86	−1.41	−2.74	1.02	−3.22	1.49
20	−1.08	−1.89	−2.86	0.69	−3.22	1.05	−0.50	−0.88	−2.28	1.27	−2.65	1.64	−1.38	−1.83	−3.70	0.94	−4.29	1.53
21	0.54	1.62	−0.49	1.56	−0.70	1.77	0.58	1.75	−0.44	1.61	−0.66	1.82	0.24	0.76	−0.74	1.23	−0.99	1.48
22	−0.17	−0.56	−1.14	0.79	−1.34	0.99	−3.14	−10.09	−4.10	−2.17	−4.30	−1.97	−0.47	−1.52	−1.42	0.48	−1.66	0.72
23	0.72	1.35	−0.93	2.37	−1.27	2.71	1.05	1.97	−0.60	2.70	−0.95	3.05	0.43	0.62	−1.70	2.55	−2.24	3.09
24	0.57	2.32	−0.19	1.32	−0.35	1.48	3.79	15.52	3.03	4.55	2.87	4.71	0.27	0.86	−0.70	1.25	−0.95	1.49
25	0.45	1.16	−0.75	1.65	−1.00	1.90	3.31	8.52	2.11	4.52	1.85	4.77	−0.16	−2.19	−0.39	0.07	−0.45	0.13
26	0.30	0.72	−1.00	1.61	−1.27	1.88	2.80	6.63	1.49	4.11	1.21	4.38	−0.31	−2.55	−0.69	0.07	−0.78	0.16
27	−0.92	−1.93	−2.39	0.56	−2.70	0.86	−2.17	−4.56	−3.65	−0.70	−3.96	−0.38	−1.53	−2.05	−3.84	0.78	−4.43	1.37
28	−0.76	−1.40	−2.44	0.92	−2.78	1.26	−5.50	−10.16	−7.18	−3.83	−7.54	−3.47	−1.37	−1.54	−4.13	1.38	−4.83	2.08
29	0.59	2.64	−0.10	1.28	−0.25	1.43	2.05	9.18	1.36	2.75	1.21	2.89	−0.02	−0.08	−0.95	0.90	−1.18	1.13
30	0.33	0.86	−0.87	1.53	−1.11	1.78	−0.49	−1.26	−1.68	0.71	−1.94	0.96	−0.28	−0.39	−2.50	1.93	−3.07	2.50
31	0.11	0.52	−0.55	0.78	−0.69	0.92	0.45	2.08	−0.22	1.11	−0.36	1.26	0.40	1.87	−0.26	1.06	−0.43	1.23
32	−0.22	−0.93	−0.94	0.50	−1.09	0.65	1.07	4.59	0.35	1.79	0.20	1.94	0.07	0.37	−0.51	0.65	−0.66	0.80
33	0.31	0.96	−0.69	1.31	−0.89	1.52	−2.18	−6.77	−3.18	−1.18	−3.40	−0.97	0.60	1.52	−0.62	1.82	−0.93	2.12
34	0.22	0.81	−0.61	1.05	−0.79	1.22	−3.10	−11.54	−3.93	−2.27	−4.11	−2.09	0.51	1.60	−0.47	1.48	−0.72	1.73
35	−0.43	−1.81	−1.16	0.31	−1.32	0.46	0.85	3.59	0.12	1.59	−0.04	1.75	−0.14	−0.50	−1.00	0.72	−1.22	0.94
36	0.01	0.06	−0.41	0.42	−0.49	0.51	2.91	21.80	2.50	3.33	2.41	3.41	0.30	2.02	−0.16	0.75	−0.27	0.86
	$T_0=2.63$						$T_0=21.80$						$T_0=2.90$

Here, $\ell =1,\dots ,36$ corresponds to each of the 36 contrasts tested by the hypothesis of no interaction, region or group as given in Equation (2) or Equation (16), respectively. In addition, ${\widehat{\delta }}_{\ell }$ is the estimated contrast, $T_{\ell }$ the value of the t-test type statistic and 95%-L or 95%-U the 95% lower or upper bound of the 95% simultaneous confidence intervals. The estimated quantiles $z_{95\%}(\max )$ are given in the headers. Different test decisions between the competing methods are marked boldface.

First, for all of the three different testing problems, the estimated quantiles of the maximum statistic $z_{95\%}(\max )$ are way larger using the randomization approach than with the simulation-based method. This is not surprising when reflecting the liberal behavior of the test. The simultaneous confidence intervals are therefore wider using the randomization procedure. In the following, results obtained for each of the three multiple null hypotheses will be discussed separately. Neither of the two competing methods detects an interaction between group and region under any of the six investigated proteins. The estimated quantiles differ remarkably, though (3.10 vs. 3.76). But, since the maximum t-statistic is $T_0=2.63$ and thus $T_0<z_{95\%}(\max )$, data do not provide the evidence to reject the null hypothesis at 5% level of significance. Investigating differences in the regions, the approximation methods provide different local conclusions at 5% level of significance. The simulation-based method declares the regions ACB under Syntaxin, M1 under VAMP2 as well as M1 under Alpha-synuclein significantly different from the average of the others, while the randomization method does not. Taking a look at the boxplots give the impression that these values differ only slightly from the mean of the others. Clearly, given the amount of regional deviations, those regions differ significantly on a pairwise level. Also, overall, the protein abundances differ significantly across the regions. Investigating differences between the groups, no significant differences can be detected using any of the competing methods. It should be noted that the estimated quantile using the randomization method increases from 3.75 to a value of 3.88, while the simulation-based estimator is still about 3.10 (3.09).

7 Discussion and outlook

Research experiments in translational medicine and especially in preclinical areas are usually small due to ethical reasons and animal welfare. Clearly, animal studies should be abandoned, but, roughly speaking, medical research has not arrived at the point yet to replace and refine every experiment to avoid animals. Often, sample sizes of such trials are smaller than 20 per experimental group, which might be a reason to argue the quality of the outcome. However, since animals are kept under homogeneous conditions, heteroscedasticity across the animals is usually smaller compared to other scenarios in humans, depending on the outcome measures. Anyway, since preclinical studies play a significant role in medical sciences in terms of transferring the results towards the next phase, a major concern is the quality of the used statistical methods. Most of them control the type-1 error rate accurately with large sample sizes only and, indeed, they show a very liberal or conservative behavior when sample sizes are small. This observation holds for a variety of statistical procedures designed for different questions and fields, including analysis of variance methods^24,37 as well as multiple contrast test procedures using maximum t-test type statistics^38,39 for repeated measures and multivariate data. When the number of comparisons is “small” compared to the sample sizes, approximate and exact methods are available.^19,40–43 These methods are, however, limited to the number of comparisons to be made and are not applicable in high-dimensional situations. Note that the methods are not applicable because of the test statistic itself (maximum t-test) or because of any computational difficulty, and information about its distribution is only available for low-dimensional designs. Recently, Chang et al.¹⁴ tackled the problem and proposed a simulation-based algorithm to approximate the distribution of the maximum statistic in high-dimensional designs. Extensive simulations show, however, that large sample sizes are needed for an accurate type-1 error rate control making their method not applicable for trials with small sample sizes. In the present paper, we modified their strategy towards a robust randomization technique to approximate the null distribution of the max t-test. Simulation studies indicate that the method approximates the null distribution very satisfactorily and greatly improves the applicability of their method.

Furthermore, the power of the method can be considerably improved by adapting it to a two-step screening procedure. For a given significance level α, define the index set

$${\mathcal{S}}_1=\{1\le \ell \le q:|T_{\ell }|\le \sqrt{2\log (q)}+{\{2\log (q)\}}^{-1/2}+\sqrt{2\log (1/(1-\alpha ))}\}$$ (17)

which contains the indices of all test statistics $T_{\ell }$ that—in absolute value—do not exceed the given bound in ${\mathcal{S}}_1$. If the cardinality of the set is equal to q, the null hypothesis $H_0:{\mathit{\theta }}_1={\mathit{\theta }}_2$ is not rejected. Otherwise, if its cardinality is equal to s (say) and smaller than q (s < q), select all corresponding test statistics in the sub-vector $\tilde{\mathit{T}}=\{T_s,s\cancel{\in }{\mathcal{S}}_1\}$. The null hypothesis will be rejected, if $\max \left\{\right|T_s|,s\cancel{\in }{\mathcal{S}}_1\}\ge y_{1-\alpha }^{*}({\mathcal{S}}_1)$. Here, $y_{1-\alpha }^{*}({\mathcal{S}}_1)$ denotes the $(1-\alpha )$-quantile of the values ${\max }_{s\cancel{\in }{\mathcal{S}}_1}\left|Y_{1s}^{*}\right|,\dots ,{\max }_{s\cancel{\in }{\mathcal{S}}_1}\left|Y_{Ms}^{*}\right|$, where the $Y^{*}$’s are defined in equation (10). Thus, the dimension of the testing problem is basically reduced to testing the null hypotheses $H_0^{(s)}:{\tilde{\mathit{\theta }}}_1={\tilde{\mathit{\theta }}}_2$, where the hypothesis matrix $\tilde{\mathit{C}}$ has appropriate dimensions and $\tilde{\mathit{\mu }}$ collects all corresponding μ_i’s being excluded from ${\mathcal{S}}_1$. Moreover, the dimension reduction implies that the critical value of the screening modification does not exceed the original one, i.e. $y_{1-\alpha }^{*}({\mathcal{S}}_1)\le y_{1-\alpha }^{*}$. As the value of the test statistic does not change ($\max \left\{\right|T_s|,s\cancel{\in }{\mathcal{S}}_1\}=T_0$ by definition), screening indeed improves the power. For the same reason, however, screening might result in even more liberal test decisions than the original version without screening. In addition, the interpretation of the screening results may be challenging in case of arbitrary contrasts or especially interaction effects. For these two reasons, we did not consider an additional screening stage. Detailed power investigations and dimension reductions will be part of future research. All of the methods considered in the paper are based on means of data. Nonparametric methods, for instance based on ranks of the data, or simultaneous methods based on interpoint distances as well as investigating other combination functions than maximum,¹² will be part of further investigations as well.