A distributed multiple sample testing for massive data

ABSTRACT

When the data are stored in a distributed manner, direct application of traditional hypothesis testing procedures is often prohibitive due to communication costs and privacy concerns. This paper mainly develops and investigates a distributed two-node Kolmogorov–Smirnov hypothesis testing scheme, implemented by the divide-and-conquer strategy. In addition, this paper also provides a distributed fraud detection and a distribution-based classification for multi-node machines based on the proposed hypothesis testing scheme. The distributed fraud detection is to detect which node stores fraud data in multi-node machines and the distribution-based classification is to determine whether the multi-node distributions differ and classify different distributions. These methods can improve the accuracy of statistical inference in a distributed storage architecture. Furthermore, this paper verifies the feasibility of the proposed methods by simulation and real example studies.

1. Introduction

In recent years, statistical inference faces tremendous challenges in computation and data storage. The exceedingly large size of data often makes it impossible to store all of them on a single machine and many applications have individual agents (local governments, research labs, hospitals and smart phones) collecting data independently. Communication between agents is prohibitively expensive due to the limited bandwidth, and direct data sharing has also raised privacy and loss of ownership concerns. These constraints make it necessary to develop methodologies for distributed systems. Distributed statistical inference has received considerable attention in modern distributed computing architecture. More specifically, it covers a wide spectrum of topics including principal component analysis [9,10], M-estimation [2,3,15,20,27,34], nonparametric regression [11,25,36], Bayesian methods [13,28], confidence intervals [6,29], quantile regression [26], bootstrap [14], and so on.

However, when the proposed distributed statistical inference method is applied to actual data, the intra-node and inter-node samples need to satisfy the most basic independent and identically distributed assumption. If the actual dataset does not satisfy this basic assumption, the actual performance may not be good no matter how effective the inference method is in theory. Therefore, it is particularly important to verify this basic assumption in practical applications. For the independence assumption, it is easily satisfied under the distributed storage architecture. For the identical distribution assumption, one needs to determine whether all node datasets come from the same population, that is, test the equality of inter-node distributions. When the sample size of each node is not large and all node samples can be easily accessed, we can adopt classical nonparametric hypothesis testing schemes to verify the identical distribution assumption. For example, when the number of node k = 2, the two-sample Kolmogorov–Smirnov (KS) test can be used to determine whether the two underlying distributions differ. However, this classical nonparametric testing is hard to conduct when each node dataset is extremely large or is not available due to the privacy protection. For instance, observations of socio-economic indicators are in general distributed and stored in various administrative regions. If we are interested in making inference on some important indicators such as lifetime and happiness index nationally, it is necessary to test whether samples in different regions come from the same distribution. But the conventional KS test is tricky in this case because the dataset in each region is usually huge and could hardly be processed in whole by a proxy simultaneously. Therefore, this paper focuses on developing distributed KS testing schemes via the divide-and-conquer strategy to solve the two-node distribution testing problem.

Our proposed testing schemes, respectively, aggregate the test decision results and p-values of hypothesis testing in each data block to obtain a final decision: whether or not to reject the null hypothesis. In general, one needs to consider what weight should be used to combine the results from each data block; however, our proposed approaches can avoid this matter. Furthermore, based on the distributed testing schemes, we also provide the fraud detection and distributed-classification approaches for multi-node machines.

The rest of this paper is organized as follows. Section 2 proposes the distributed multiple sample testing – the distributed two-node KS testing schemes. In Section 3, we conduct some discussions on block size in distributed test schemes. Section 4 provides a fraud detection approach in the distributed architecture via the distributed KS two-node testing schemes. Section 5 discusses a classification method to test whether the multi-node distributions differ and classify different distributions. Section 6 applies the proposed testing schemes to real data. We conclude in Section 7. When appropriate, detailed proofs are relegated to the appendix.

2. The distributed two-node KS testing schemes

2.1. Problem setup

When the sample stored in two nodes is relatively small, we can compute KS test statistic on a proxy easily and quickly. However, it can be prohibitive to transmit or deal with full data by the proxy due to bandwidth limitation or memory constraint when the amount of data stored in two nodes is huge. In this setting, the conventional KS test procedure is difficult to implement. Therefore, we propose the distributed two-node KS testing schemes via the divide-and-conquer strategy: batch test and then aggregation. The proposed testing schemes not only solve the problem that the conventional test cannot be directly used, but also improve the computation efficiency as shown in the later discussion. For simplicity, we assume the sample sizes of the two-node machines are identical. For details on the two-sample KS test, see [1,18,19,23].

2.2. The testing schemes

There are N independent samples $\{X,Y\}$ from continuous distribution functions F and G, and they are stored in two-node machines. We randomly and evenly partition these data into m subsets $\{X^{\left(i\right)},Y^{\left(i\right)}{\}}_{i=1}^m$ with a sample size of n where $N=m\times n$ and n is manageable. Our goal is to test

$$H_0:F=G,\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{u}\mathrm{s}{H}_1:F\ne G.$$ (1)

At first, we conduct the two-sample KS test for each data block $\{X^{\left(i\right)},Y^{\left(i\right)}\},i=1,\dots ,m$. More specifically, the two-sided KS test statistic for the ith data block is

$$D_n^{\left(i\right)}=\sqrt{n/2}\underset{u}{sup}|F_n^{\left(i\right)}\left(u\right)-G_n^{\left(i\right)}\left(u\right)|,$$

where $F_n^{\left(i\right)}\left(u\right)$ and $G_n^{\left(i\right)}\left(u\right)$ are the empirical distribution functions of the $X^{\left(i\right)}$ and $Y^{\left(i\right)}$, respectively. According to [24], we have

$$\underset{n\to \mathrm{\infty }}{lim}P(D_n^{\left(i\right)}\le z|H_0)=K\left(z\right),z>0,$$

where

$$K\left(z\right)=1-2[{\mathrm{e}}^{-2z^2}-{\mathrm{e}}^{-2(2z{)}^2}+{\mathrm{e}}^{-2(3z{)}^2}-\cdots ]=1-2\sum _{k=1}^{+\mathrm{\infty }}(-1{)}^{k-1}{\mathrm{e}}^{-2k^2{z}^2}.$$

Then we reject $H_0$ with the significant level α for the ith data block if

$$D_n^{\left(i\right)}>K_{1-\alpha },$$

where $K_{1-\alpha }$ is the $(1-\alpha )$th quantile of the Kolmogorov distribution $K\left(z\right)$.

The asymptotic analysis of the distribution of a two-sample Smirnov statistic and the rate of convergence in the distribution of a two-sided Smirnov statistic were studied extensively in [5,8,12]. We have the following proposition.

Proposition 2.1

For any $z>0,$ as $n\to \mathrm{\infty },$ there holds

$$\underset{z}{sup}|P(D_n^{\left(i\right)}\le z|H_0)-K\left(z\right)|=O\left(n^{-1/2}\right),$$

where $i=1,2,\dots ,m$.

Next, we will, respectively, aggregate the test decision results and p-values of hypothesis testing in each data block to obtain a final decision.

• Scheme I

Assume that $h_n^{\left(i\right)}$ is the ith test decision result with the significance level α under the null hypothesis $H_0$, $i=1,2,\dots ,m$, where $h_n^{\left(i\right)}$ is defined as

$$h_n^{\left(i\right)}=\left\{\begin{array}{ll}1,& \mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0;\\ 0,& \mathrm{a}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{p}\mathrm{t}{H}_0.\end{array}\right.$$ (2)

Furthermore, we have that

$$h_n^{\left(1\right)}|H_0,h_n^{\left(2\right)}|H_0,\dots ,h_n^{\left(m\right)}|H_0\sim \mathrm{i}.\mathrm{i}.\mathrm{d}.\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left({\alpha }_n\right).$$

where ${\alpha }_n=P(h_n^{\left(i\right)}=1|H_0)$. Based on $h_n^{\left(i\right)},i=1,2,\dots ,m$, we now turn to consider the following testing problem

$$H_0^a:{\alpha }_n\le \alpha ,\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{u}\mathrm{s}{H}_1^a:{\alpha }_n>\alpha .$$ (3)

To this end, we construct the test statistic

$$U_{m,n}^a=\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m\alpha }{\sqrt{m\alpha (1-\alpha )}}.$$

To conduct the testing, we need to know the limiting distribution of the constructed statistic. We establish the asymptotic distribution of the test statistic $U_{m,n}^a$ in the following theorem.

Theorem 2.2

Provided that $m=o\left(n\right),$ under the null hypothesis $H_0,$ we have

$$\underset{m,n\to \mathrm{\infty }}{lim}P(U_{m,n}^a\le t)=\mathrm{\Phi }\left(t\right),$$ (4)

where $\mathrm{\Phi }\left(t\right)$ is the cumulative distribution function of the standard normal distribution.

Thus, the testing scheme for (3) is

$$\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0^a,\mathrm{i}\mathrm{f}{U}_{m,n}^a>Z_{1-\alpha },$$

where $Z_{1-\alpha }$ is $(1-\alpha )$th quantile of the standard normal distribution.

This testing procedure aggregates m test decision results and thus produces a reasonable distributed testing scheme: if $H_0^a$ is rejected, $H_0$ is accordingly rejected as well, that is,

$$\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0,\mathrm{i}\mathrm{f}{U}_{m,n}^a>Z_{1-\alpha }.$$

For simplicity, this distributed testing scheme is called the DCR-KS test.

• Scheme II

In addition to the above test decision results, we can also obtain the corresponding p-values, denoted as $p_n^{\left(i\right)}$, when the KS test for the ith data block is implemented. Let G be the asymptotic distribution function of $p_n^{\left(i\right)}$ for any i, then the function G is a uniform distribution under the null hypothesis. Hence, we now turn to consider the following hypothesis testing problem

$$H_0^b:G=G_0,\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{u}\mathrm{s}{H}_1^b:G\ne G_0,$$

where $G_0$ is the cumulative function of the standard uniform distribution.

To this end, we construct the Cramér–Von Mises-type test statistics

$$U_{m,n}^b=m{\int }_0^1\left(G_{m,n}\right(x)-G_0(x\left){)}^2\mathrm{d}{G}_0\right(x),$$

where $G_{m,n}\left(x\right)$ is the empirical distribution function of $\{p_n^{\left(i\right)}{\}}_{i=1}^m$.

It can be inferred that under the null hypothesis $H_0$, when $m=o\left(n^{1/2}\right)$ we have

$$U_{m,n}^b\stackrel{d}{⟶}W\triangleq {\int }_0^1{B}^2\left(u\right)\mathrm{d}u,\mathrm{a}\mathrm{s}m,n\to \mathrm{\infty },$$ (5)

where $B\left(u\right)$ is the standard Brownian bridge. In this case, the testing scheme can be given by

$$\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0,\mathrm{i}\mathrm{f}{U}_{m,n}^b>C_{1-\alpha },$$

where $C_{1-\alpha }$ is $(1-\alpha )$th quantile of the distribution of the asymptotic distribution of the test statistic $U_{m,n}^b$. This testing scheme is called the DCP-KS test.

Remark 2.1

It is worth noting that the critical values used in the two schemes are determined by the corresponding limiting null distributions whose theoretical bases are, respectively, established under the conditions of $m\to \mathrm{\infty },m=o\left(n\right)$ and $m\to \mathrm{\infty },m=o\left(n^{1/2}\right)$. Therefore, m cannot be too small for DCP-KS and DCR-KS in practice.

Next, we simply discuss the lower bound of m for DCR-KS and DCP-KS in applications. Essentially, the DCR-KS aggregation testing problem is a large sample Bernoulli proportion hypothesis testing. For this testing problem, the sampling distribution of the proportion can be well approximated by a normal distribution under $H_0$ when $m\alpha >5$ with a significance level of α (see [30]). Similarly, the DCP-KS aggregation problem is a goodness-of-fit hypothesis testing. For this testing problem, when the Cramér–von Mises test is adopted, the sampling distribution of the statistic can be well approximated by the asymptotic distribution with the sample size greater than 10 (see [7]). Hence, with a significance level of $5\mathrm{\%}$, we suggest to select m>100 for DCR-KS and m>10 for DCP-KS.

3. Discussion of block size m

It is a basic problem on the choice of m in the application of a distributed algorithm. The general purpose of selecting m is to balance the statistical efficiency and the computational cost – that is, it is desirable to reduce the computational cost as much as possible without losing the statistical efficiency. This problem has been paid more and more attention by researchers in recent years. Here, we also discuss some related problems on block size for the proposed testing schemes.

3.1. The effect of block size on the test performance

We first investigate the performance of the proposed DCR-KS and DCP-KS tests with different m via a simulation study. Test size and power are, respectively, calculated based on 1000 independent trials. The two-node samples are independently generated from the normal distribution with $X\sim N(0,1)$ and $Y\sim N(\delta ,1),$ where $\delta =0,0.03,0.04,0.05$. In particular, $\delta =0$ corresponds to true $H_0$ which will be used to examine the size of the test statistics. The results are summarized in Figure 1 with a sample size of $N={10}^5$.

Figure 1.

(a) Size for the two testing schemes as a function of block size. Each method is shown using a different colored symbol; (b) power with mean increment $\delta =0.03,0.04$ and 0.05 for the two testing schemes as a function of block size. Each increment is shown using a different colored symbol.

Panel (a) of Figure 1 displays the sizes of DCR-KS and DCP-KS. It can be seen that the sizes of the two tests are close to both the nominal level $\alpha =0.05$ and the level of oracle with the full sample when $m\in [40,200]$ for DCR-KS and $m\in [10,50]$ for DCP-KS. Overall, DCP-KS performs better for small m, but DCR-KS performs better for large m in controlling type I error. Besides, panel (b) of Figure 1 displays the powers of the two tests for three different mean increments $\delta =0.03$, 0.04 and 0.05. We see that both DCR-KS and DCP-KS test powers increase as δ goes from 0.03 to 0.05 but decline as m grows. It is clear from the results that the block size does affect the performance of the two tests, and therefore, an appropriate block size is desirable in order to achieve a reasonable size and power of test.

In addition, we also compare computation time between the proposed testing schemes and the conventional testing scheme when massive data could be handled by a single machine. In the setting with a sample size of ${10}^8$ from the null distribution $N(0,1)$, we record the computation time of three testing schemes on a desktop computer with a 3.20 GHz Inter Core i7-8700 CPU and 16 GB memory as shown in Table 1. The result shows that the DCR-KS and DCP-KS tests can save up to 70% of time compared with the conventional testing scheme, which reflects that our schemes can greatly improve computation efficiency for massive data in a limited memory constraint.

Table 1.

The computation time of the three testing schemes.

Case	Method	Time
$N={10}^8$	DCR-KS	63.5060 s
m = 100	DCP-KS	64.3648 s
$N(0,1)$vs. $N(0,1)$	Oracle	237.2067 s

3.2. A data-driven selection criterion on block size

Another interesting problem is how to select m in the application of DCR-KS and DCP-KS testing schemes. Some data-driven methods on the choice of m have been proposed in the divide-and-conquer framework. For example, Shang and Cheng [21] found that the block size could be viewed as an alternative regularization parameter in distributed nonparametric regression. Later, Xu et al. [32] proposed a cross-validation approach for selecting the tuning parameter in distributed nonparametric learning.

Inspired by Shang and Cheng [21] and Xu et al. [32], we also propose a data-driven criterion named BS that can be used to select the block size. Specifically, the proposed BS criterion is formulated as the following optimization problem:

$$\underset{m}{min}\left\{\sum _{i=1}^m(D_n^{\left(i\right)}-\overline{D_n}{)}^2+\lambda \ast \log (1/m)\right\},$$ (6)

where $D_n^{\left(i\right)}$ is the individual test statistic, $\overline{D_n}$ is the average of test statistics and λ is a tuning parameter related to the full sample size.

The first term of (4) represents the volatility of the estimated test statistic of each block and the second term represents the computation cost. The volatility between each estimator will increase, but the overall computational cost will decrease as m grows. In general, we can select $\lambda =\log \left(N\right)$. Figure 2 displays the objective function of (4) versus m based on 100 independent trials under null distribution with $N(0,1)$. It can be seen that the objection function starts to decrease and then increases from some point as the block size increases. Therefore, an optimal m can be obtained in order to balance the test performance and computational cost.

Figure 2.

The relationship between the objective function and m in different sample sizes.

Next, we will check the performance of the BS criterion via simulation studies. For DCR-KS, it holds that the asymptotic validity of a test can be achieved with $m<N^{1/2}$. Besides, m is in general no less than 10 considering that the null distribution of test statistics in aggregation is derived asymptotically. Hence, we select m for DCR-KS in the range of $\{m|m=N^r,r\in (0,0.5\left)\right\}$. Similarly, we select m for DCP-KS in the range of $\{m|m=N^r,r\in (0,0.4\left)\right\}$. The two-node samples are independently generated from underlying distributions being Normal, Uniform and Gamma, respectively. Size and power are calculated as the proportions of rejection based on 1000 independent trials under the significant level of $\alpha =5\mathrm{\%}$. The results are summarized in Tables 2 and 3.

Table 2.

Estimated probability of type $\mathbf{I}$ error for sample size N.

	$N(0,1)$			$U(0,1)$			$\mathrm{G}\mathrm{a}(1/2,1/2)$
N	DCR-KS	DCP-KS	Oracle ${}^{\mathrm{a}}$	DCR-KS	DCP-KS	Oracle	DCR-KS	DCP-KS	Oracle
${10}^5$	0.056	0.055	0.058	0.056	0.047	0.059	0.052	0.053	0.054
${10}^6$	0.049	0.051	0.052	0.048	0.043	0.044	0.048	0.048	0.055
${10}^7$	0.054	0.050	0.057	0.049	0.053	0.052	0.056	0.045	0.052
${10}^8$	0.052	0.051	0.047	0.052	0.048	0.052	0.052	0.047	0.048

${}^{\mathrm{a}}$The results of Oracle correspond to those based on the full sample.

Table 3.

Power of the DCR-KS test (the DCP-KS test is in the parenthesis) for different sample sizes under normal distribution.

Distribution class	100,000	200,000	400,000	600,000	800,000	1,000,000	Oracle ${}^{\mathrm{a}}$
Different mean and same variance
$N(0,1)$vs. $N(0.010,1)$	0.08 (0.06)	0.12 (0.11)	0.17(0.23)	0.19 (0.35)	0.25 (0.51)	0.36 (0.66)	1
$N(0,1)$vs. $N(0.015,1)$	0.13 (0.10)	0.23 (0.31)	0.40 (0.69)	0.54 (0.90)	0.67 (0.98)	0.81 (1.00)	1
$N(0,1)$vs. $N(0.020,1)$	0.19 (0.25)	0.39 (0.66)	0.74 (0.97)	0.90 (0.99)	0.97 (1.00)	0.98 (1.00)	1
$N(0,1)$vs. $N(0.025,1)$	0.33 (0.52)	0.68 (0.92)	0.94 (1.00)	0.99 (1.00)	1.00 (1.00)	1.00 (1.00)	1
$N(0,1)$vs. $N(0.030,1)$	0.49 (0.78)	0.86 (0.99)	0.99 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1
Same mean and different variance
$N(0,1)$vs. $N(0,{1.02}^2)$	0.06 (0.09)	0.12 (0.31)	0.15 (0.72)	0.20 (0.93)	0.23 (0.99)	0.30 (0.99)	1
$N(0,1)$vs. $N(0,{1.04}^2)$	0.18 (0.80)	0.37 (0.99)	0.71 (1.00)	0.90 (1.00)	0.97 (1.00)	0.99 (1.00)	1
$N(0,1)$vs. $N(0,{1.06}^2)$	0.44 (1.00)	0.88 (1.00)	0.99 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1
Different mean and variance
$N(0,1)$vs. $N(0.01,{1.02}^2)$	0.12 (0.17)	0.18 (0.53)	0.32 (0.92)	0.45 (0.98)	0.60 (0.99)	0.73 (1.00)	1
$N(0,1)$vs. $N(0.02,{1.04}^2)$	0.44 (0.98)	0.81 (1.00)	0.99 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1
$N(0,1)$vs. $N(0.03,{1.06}^2)$	0.92 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1.00 (1.00)	1

${}^{\mathrm{a}}$The results of Oracle correspond to those based on full sample.

Table 2 shows the test size for two-node samples taken from $N(0,1)$, $U(0,1)$ and $\mathrm{G}\mathrm{a}(1/2,1/2)$. We see that the two testing schemes can control the type $\mathbf{I}$ error for three different distributions, and the empirical sizes are closer to both the nominal level $\alpha =0.05$ and the level of oracle with the full sample. Table 3 only shows the power performance of the two testing schemes when the two-node samples come from $F\sim N({\mu }_1,{\sigma }_1^2)$ and $G\sim N({\mu }_2,{\sigma }_2^2)$ with various values of $({\mu }_1,{\sigma }_1,{\mu }_2,{\sigma }_2)$. It can be seen that the test power improves as the sample size increases and the difference between F and G increases. Similar results are obtained by the Uniform and Gamma cases.

The above results not only illustrate the feasibility of the BS criterion but also reflect the effectiveness of our testing schemes. In addition, we find that the selected average block size is around $N^{0.4}$ for DCR-KS and $N^{0.3}$ for DCP-KS. Therefore, we can also choose an empirical block size $m\approx N^{0.4}$ for DCR-KS or $m\approx N^{0.3}$ for DCP-KS instead of the BS criterion in practice.

Remark 3.1

In practice, the massive data are recommended to be divided into a small number of blocks for some highly configured computing equipment. However, we can conclude in the discussion of Remark 2.1 that DCR-KS is not suitable for small m, while DCP-KS can be applied when m is not too small such as greater than 10. Here, we develop another testing scheme for tiny m ( $m\le 10$), called DCS-KS. The key idea of DCS-KS is that the sum of m block test statistics is asymptotically distributed as the mth order convolution of the Kolmogorov distribution under $H_0$ for large N and tiny m. Specifically, the DCS-KS test statistic is

$$U_{m,n}^{\mathrm{D}\mathrm{C}\mathrm{S}}=\sum _{i=1}^m{D}_n^{\left(i\right)},$$

where $D_n^{\left(i\right)}$ is the two-sided KS test statistic for ith data block, $i=1,2,\dots ,m$. Then, the testing scheme is given by

$$\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0,\mathrm{i}\mathrm{f}{U}_{m,n}^{\mathrm{D}\mathrm{C}\mathrm{S}}>L_{1-\alpha }^m,$$

where $L_{1-\alpha }^m$ is the $(1-\alpha )$th quantile of $\underset{m}{\underbrace{K\ast K\ast \dots \ast K}}\left(z\right)$ that is the mth order convolution function of the Kolmogorov distribution. The explicit expression of $K\ast K\left(z\right)$ is given when m = 2 in Appendix 3. We can get the approximation of the critical value $L_{1-\alpha }^m$ via the Monte Carlo method for m>2.

Similarly, we assess the performance of the DCS-KS testing scheme for tiny m with sample size $N={10}^5$. The results are summarized in Tables 4 and 5. It can be seen that DCS-KS performs ideally in terms of test size and power for tiny m. Specifically, the size of the DCR-KS test is close to nominal level 0.05 for all cases and is less sensitive to the choice of m for tiny m. In addition, the power of the DCR-KS test increases and is closer to that of oracle with full sample as the distribution difference increases. Therefore, the DCS-KS is an efficient testing scheme for tiny m.

Table 4.

The estimated probability of type $\mathbf{I}$ error of the DCS-KS test for different block sizes.

	DCS-KS
m	$N(0,1)$	$U(0,1)$	Gamma(1/2,1/2)
2	0.052	0.049	0.050
4	0.048	0.051	0.050
5	0.058	0.050	0.053
8	0.054	0.052	0.057
10	0.052	0.054	0.056

Table 5.

Power of the DCS-KS test for different block sizes under normal distribution.

m	2	4	5	8	10	Oracle ${}^{\mathrm{a}}$
$N(0,1)$vs. $N(0.01,1)$	0.45	0.36	0.30	0.23	0.21	1
$N(0,1)$vs. $N(0.02,1)$	0.97	0.92	0.90	0.83	0.79	1
$N(0,1)$vs. $N(0.03,1)$	1.00	1.00	1.00	1.00	0.99	1

${}^{\mathrm{a}}$The results of Oracle correspond to those based on full sample dataset.

According to the above discussion, we give some suggestions in applications. (a) For some highly configured computing equipment, the massive data are recommended to be divided into a small number of blocks ( $2\le m\le 10$) and we strongly recommend to use of the DCS-KS testing scheme. (b) DCR-KS and DCP-KS are useful for general computing equipment. It is worth noting that DCR-KS is not suitable for small m. If one needs to adopt the aggregation strategy on test decision for small m, the likelihood ratio test for Bernoulli observations can be used, but this testing scheme is conservative in terms of test size (see Appendix 4).

In the following topics, we only provide some new methods based on the DCR-KS testing scheme, and some related studies based on the DCP-KS testing scheme are similar.

4. Fraud detection

When the data are stored in a distributed manner, we often solve statistical problems via the divide-and-conquer technique, which is based on the assumption that the distributed node datasets come from the same underlying distribution F. However, there might be some node datasets which contain fraud samples from other distributions. Therefore, in order to improve the efficiency of the distributed estimation, we need to detect which node contains fraud data. We will study this problem based on the proposed testing schemes in a distributed architecture.

4.1. Algorithm

In a distributed architecture, there are s + 1 node machines where the s + 1 massive datasets $\{X,Y_1,Y_2,\dots ,Y_s\}$ are stored. And the dataset X is a true sample from the underlying continuous distribution F. Our goal is to detect which node contains fraud data in the remaining s nodes. This detection problem can be transformed to test whether the remaining s datasets come from the same distribution as the true dataset X. To this end, we perform s hypothesis testing procedures on the equivalence of two distributions via the DCR-KS testing scheme (see Algorithm 1).

Based on the above procedure, we can conclude that datasets in $C_2$ have different distributions from the dataset X, which indicates that there are some fraud data in these datasets. Therefore, before conducting statistical inference in distributed structure with s + 1 node machines, we should pick out these fraud datasets to improve the efficiency of inference.

4.2. Simulation studies

In this subsection, we will use simulation to discuss the performance of fraud node detection with the DCR-KS test in the distributed structure and the effect of fraud detection on the distributed statistical estimation

In this simulation, there are four nodes where we generate the true node dataset X from $N(0,1)$ and the other three-node datasets $\{Y_1,Y_2,Y_3\}$ are generated from $N(\mu ,{\sigma }^2)$ with various values of $(\mu ,\sigma )$. We conduct s = 3 hypothesis tests via DCR-KS approach with nominal level $\alpha =5\mathrm{\%}$. All node datasets have the same sample N for each dataset. We evaluate the performance of fraud detection in terms of classification accuracy, which is the proportion of tested datasets where it is classified correctly. In addition, we compute the overall mean estimators based on DCR-KS detection results, the one-shot distributed approach [34] and sample dataset X, which are denoted as DCM-KS, DCM and M. Here, the one-shot distributed approach refers to the average individual mean estimators for all node datasets. Hence, we evaluate the effect of fraud detection on the distributed statistical estimation in comparison to DCM-KS, DCM and M. All simulations are conducted with 1000 replicates, and the simulation results are listed in Table 6.

Table 6.

Classification accuracy and mean estimation performance with true node dataset $X\sim N(0,1)$.

$Y_1$	$Y_2$	$Y_3$	Accuracy	DCM	DCM-KS	M
$N=\mathrm{100,000}$
$N(0,1)$	$N(0,1)$	$N(0,1)$	94.23%	−0.00004	0.00002	−0.0001
$N(0,1)$	$N(0,1)$	$N(0.05,1)$	95.37%	0.0125	0.0003
$N(0,1)$	$N(0.05,1)$	$N(0.05,1)$	96.53%	0.0249	0.0007
$N(0.04,1)$	$N(0.05,1)$	$N(0.06,1)$	92.67%	0.0374	0.0037
$N=\mathrm{200,000}$
$N(0,1)$	$N(0,1)$	$N(0,1)$	95.67%	−0.00006	−0.00006	−0.00007
$N(0,1)$	$N(0,1)$	$N(0.05,1)$	97.07%	0.0125	0.00003
$N(0,1)$	$N(0.05,1)$	$N(0.05,1)$	98.47%	0.0250	−0.00004
$N(0.04,1)$	$N(0.05,1)$	$N(0.06,1)$	97.13%	0.0375	0.0016

From the perspective of classification accuracy, the classification accuracy reaches more than 95% for common cases and the classification error rate can be kept within 10% for the extreme two cases. The classification accuracy will improve as the sample size increases in all cases. In addition, from the estimation results of the overall mean, the performance of estimation is becoming better as the same distributed nodes are increasing and accuracy is becoming worse as the number of nodes with false distributions is increasing for all cases. It is also seen that the DCM-KS is better than the DCM for different distribution cases, which explains the importance of fraud node detection in distributed statistical inference. These results reveal the feasibility and efficiency of the proposed DCR-KS test for fraud detection.

5. The distribution-based classification for multi-nodes

In the previous section, we have studied the fraud detection in a distributed architecture, which is equivalent to the binary classification problem based on the prior information of some real data. In this section, we will discuss the distribution-based node classification problem in a distributed architecture, that is, categorizing the node dataset with the same distribution into one category. The study of the classification approach can not only test whether the multi-node distributions differ but also classify different distributions, which contributes to the subsequent statistical analysis in a distributed architecture.

5.1. The distribution-based classification algorithm for multi-nodes

In a distributed architecture, there are s node machines that store the s massive datasets $\{Y_1,Y_2,\dots ,Y_s\}$ from underlying continuous distribution $\{F_1,F_2,\dots ,F_s\}$, where there exists $F_i\ne F_j$ for some $i\ne j$. Our goal is to cluster datasets with the same distribution and divide the s datasets into different categories. In this regard, we propose a classification method on distribution through the DCR-KS test, see Algorithm 2.

5.2. Simulation studies

We will conduct this classification approach with simulated data. We choose s = 4 and the same sample size $N=\mathrm{100,000}$, and discuss the following five cases.

• Case I: $Y_i\sim N(0,1)$, i = 1, 2, 3, 4

• Case II: $Y_i\sim N(0,1)$, i = 1, 2, 3, $Y_4\sim N(0.05,1)$

• Case III: $Y_i\sim N(0,1)$, i = 1, 2, $Y_j\sim N(0.05,1)$, j = 3, 4

• Case IV: $Y_i\sim N(0,1)$, i = 1, 2, $Y_3\sim N(0.05,1)$, $Y_4\sim N(0.10,1)$

• Case V: $Y_1\sim N(0,1)$, $Y_2\sim N(0.05,1)$, $Y_3\sim N(0.10,1)$, $Y_4\sim N(0.15,1)$

In addition, we define five classification evaluation indices, that is,

• Correct Classification Number Proportion (CCNP): the proportion of the cases where the number of the estimated categories is equal to the number of real categories.

• Correct Classification Proportion(CCP): the proportion of the cases where the estimated categories are same as the real categories.

• Classification Purity (CP): the proportion of the cases where all datasets in each category come from the same distribution.

• Meterogeneous Misjudgment Proportion (MMP): the proportion of the cases where some datasets that come from the same distribution is classified into different categories.

• Heterogeneous Misjudgment Proportion (HMP): the proportion of the cases where some datasets that come from the different distribution is classified into one category.

All simulations are conducted with 1000 replicates. And the summary of simulation results is listed in Table 7.

Table 7.

Classification performance in different cases with significance level $\alpha =0.05$.

Case	Class	CCNP	CCP	CP	MMP	HMP
I	1	83.80%	83.80%	100%	16.2%	–
II	2	86.30%	84.80%	96.70%	13.4%	3.4%
III	2	89.10%	84.70%	95.30%	15.1%	15.1%
IV	3	89.50%	87.00%	93.80%	9.3%	6.2%
V	4	92.90%	92.90%	92.90%	–	7.10%

Note: ‘–’ denotes null value.

It can be seen that our proposed classification is feasible in terms of the classification evaluation indices. Overall, both the CCNP and the CCP are over 80%, and the CP is over 90% for all cases. It can also be seen from MMP and HMP that the misjudgment proportion is below 20% for all cases. Therefore, our proposed classification approach is applicable in tolerance of 20%. In addition, the accuracy of classification is higher when the difference between the datasets is larger, which indicates that the proposed classification method prefers to deal with the cases where there are significant differences between the node datasets.

6. Application

We apply the DCR-KS and DCP-KS testing schemes to analyze the Central England Temperature (CET) dataset, which is a meteorological time series dataset consisting of 246 years ( 1772–2017) of average daily temperature in Central England. The CET dataset represents the longest continuous thermometer-based temperature record on earth and was previously analyzed by Berkes et al. [4] and Zhang et al. [33] in the context of inference for functional time series. The existing studies conclude that there are two possible change points in the CET dataset, the years 1927 and 1993. For this dataset, we are interested in testing whether the distribution of the data containing the change points is the same as that of the data before the change point, which will provide new inferential evidence for the change points.

In our analysis, we treat the dataset as a univariate time series sample of daily average temperature. The sample size is $246\times 365=\mathrm{89,790}$, where we ignore leap years. We remove seasonality by subtracting each observation from the mean temperature for that calendar day across 246 years and then divide this massive dataset into three groups with the same sample size, where group II and group III, respectively, contain one change point found in the literature. The grouped datasets are shown in Table 8. We use the Gaussian kernel density estimation (KDE) and histogram to display the distributions of the three datasets. It can be seen from Figure 3 that the three distributions are slightly different.

Table 8.

Summary of the CET dataset partition.

	Group
Summary	I	II	III
Time-interval	1772–1853	1854–1935	1936–2017
Sample size N	29,930	29,930	29,930

Figure 3.

The KDE curve and histogram of the three datasets.

We apply the proposed DCR-KS and DCP-KS testing schemes to test the equality of three group distributions. Denote $F_1,F_2$ and $F_3$ as distributions of groups I, II and III. We, respectively, choose m = 73 and m = 41 as block sizes of the DCR-KS and DCP-KS testing schemes. Table 9 displays the results of DCR-KS, DCP-KS and the conventional KS test with full sample oracle at the significant level of $\alpha =5\mathrm{\%}$. Note that the results of the DCR-KS and DCP-KS test show that three null hypotheses are all significantly rejected $(p<0.0001)$, which is consistent with the results of Oracle. This example shows that the two-node distributed KS testing schemes are applicable.

Table 9.

Test result of distribution on CET three datasets with 5% level.

	Null hypothesis $H_0$
Method	$F_1=F_2$	$F_1=F_3$	$F_2=F_3$
DCR-KS	Rejected	Rejected	Rejected
DCP-KS	Rejected	Rejected	Rejected
Oracle ${}^{\mathrm{a}}$	Rejected	Rejected	Rejected

${}^{\mathrm{a}}$The results of Oracle correspond to those based on full sample dataset.

7. Conclusion

In this paper, we focus on the distributed multiple sample testing schemes for a two-node machine, called DCR-KS and DCP-KS, to test the equality of distribution in massive data. The main idea of the testing scheme is to design a proper divide-and-conquer strategy. The proposed testing schemes are shown to be able to provide feasible nonparametric testing solutions for decentralized data storage architectures and can be applied in many areas. In addition, we develop the fraud detection and the distribution-based classification methods via the proposed testing schemes which reveal the value of the distributed testing schemes.

In the divide-and-conquer framework, the aggregation of local results in general adopts simple averaging rules [16,35]. Besides, some authors have proposed the weighted averaging rules for aggregation [17,22]. The weighted rules are proposed mostly because the local results don't satisfy the identically distributed assumption in some scenarios or models, and it is necessary to weigh local results in order to achieve the asymptotic optimality. But in this paper, we adopt the simple averaging rule to conquer because it holds that each local result is independent and identically distributed under the null hypothesis so that each local result plays an equally important role in aggregation.

Although the KS test is focused on in this paper, the proposed distributed testing schemes can be easily extended to other two-sample tests due to their aggregation nature. Because the computation of the Kolmogorov distribution is a little bit complicated, two-sample tests with asymptotic null distributions computed easily are recommended, for example, the likelihood ratio-based test approach proposed by Xing et al. [31] whose limiting null distribution is the conventional chi-square.

Appendix 1. Proof of Proposition 2.1

Proof.

Let

$${\mathrm{\Delta }}_n=\underset{z>0}{sup}|P(D_n^{\left(i\right)}\le z|H_0)-K\left(z\right)|.$$

According to Theorem 2 in [5], for the ith data block, we have

$$\underset{z>0}{sup}|P(D_n^{\left(i\right)}\le z|H_0)-K\left(z\right)|\le \frac{e\pi }{2}{n}^{-1/2}+\frac{4}{\sqrt{e}}{n}^{-1},$$

then as $n\to \mathrm{\infty }$, there holds

$$\underset{z>0}{sup}|P\{D_n^{\left(i\right)}\le z|H_0\}-K\left(z\right)|=O\left(1/\sqrt{n}\right).\text{■}$$

Appendix 2. Proof of Theorem 2.2

Proof.

According to Proposition 2.1, we have

$${\alpha }_n-\alpha =O\left(1/\sqrt{n}\right),\mathrm{a}\mathrm{s}n\to \mathrm{\infty }.$$

Secondly, based on $h_n^{\left(1\right)},h_n^{\left(2\right)},\dots ,h_n^{\left(m\right)}$, from the Central Limit Theorem, we have

$$\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m{\alpha }_n}{\sqrt{m{\alpha }_n(1-{\alpha }_n)}}|H_0\stackrel{d}{⟶}N(0,1),m\to +\mathrm{\infty }.$$

Let ${\eta }_n={\alpha }_n-\alpha$, then

$$\begin{array}{rl}P\{U_{m,n}^a\le t|H_0\}& =P\left\{\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m\alpha }{\sqrt{m\alpha (1-\alpha )}}\le t|H_0\right\}\\ & =P\left\{\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m{\alpha }_n+m({\alpha }_n-\alpha )}{\sqrt{m({\alpha }_n-({\alpha }_n-\alpha \left)\right)(1-{\alpha }_n+({\alpha }_n-\alpha \left)\right)}}\le t|H_0\right\}\\ & =P\left\{\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m{\alpha }_n+m{\eta }_n}{\sqrt{m{\alpha }_n(1-{\alpha }_n)}}\le \frac{t\sqrt{m({\alpha }_n-{\eta }_n)(1-{\alpha }_n+{\eta }_n)}}{\sqrt{m{\alpha }_n(1-{\alpha }_n)}}|H_0\right\}\\ & =P\left\{\frac{\sum _{i=1}^m{h}_n^{\left(i\right)}-m{\alpha }_n}{\sqrt{m{\alpha }_n(1-{\alpha }_n)}}\le \frac{t\sqrt{m({\alpha }_n-{\eta }_n)(1-{\alpha }_n+{\eta }_n)}-m{\eta }_n}{\sqrt{m{p}_n(1-{\alpha }_n)}}|H_0\right\}\\ & ⟹m\to \mathrm{\infty }\mathrm{\Phi }\left(t\sqrt{\frac{({\alpha }_n-{\eta }_n)(1-{\alpha }_n+{\eta }_n)}{{\alpha }_n(1-{\alpha }_n)}}-\underset{m\to \mathrm{\infty }}{lim}\frac{\sqrt{m}{\eta }_n}{\sqrt{{\alpha }_n(1-{\alpha }_n)}}\right)\\ & ⟹n\to \mathrm{\infty }\mathrm{\Phi }\left(t-\underset{n,m\to \mathrm{\infty }}{lim}\frac{\sqrt{m}{\eta }_n}{\sqrt{{\alpha }_n(1-{\alpha }_n)}}\right).\end{array}$$

With $m=o\left(n\right)$, $\underset{n\to \mathrm{\infty }}{lim}{\alpha }_n=\alpha$, and ${\eta }_n=O\left(1/\sqrt{n}\right)$, we have

$$\underset{n,m\to \mathrm{\infty }}{lim}\frac{\sqrt{m}{\eta }_n}{\sqrt{{\alpha }_n(1-{\alpha }_n)}}=0,$$

that is,

$$\underset{m,n\to +\mathrm{\infty }}{lim}P(U_{m,n}^a\le t)=\mathrm{\Phi }\left(t\right).\text{■}$$

Appendix 3. The expression of $K\ast K\left(z\right)$ for m = 2

Suppose that random variables $X_1$ and $X_2$ are independent and identically distributed as $K\left(x\right)$, and

$$K\left(x\right)=1-2\sum _{k=1}^{+\mathrm{\infty }}(-1{)}^{k-1}{\mathrm{e}}^{-2k^2{x}^2},$$

where x>0. The density function of the Kolmogorov distribution is

$$f\left(x\right)=K^{\prime }\left(x\right)=8\sum _{k=1}^{+\mathrm{\infty }}\left\{\right(-1{)}^{k-1}\left(k^{2}x\right){\mathrm{e}}^{-2k^2{x}^2}\}.$$

Next, we compute the distribution of $X_1+X_2$. For any y>0, there holds

$$\begin{array}{rl}P(X_1+X_2\le y)& ={\iint }_{x_1+x_2\le y}k(x_1,x_2)\mathrm{d}{x}_1\mathrm{d}{x}_2\\ & ={\int }_0^{+\mathrm{\infty }}f\left(x\right)K(y-x)\mathrm{d}x\\ & =1-16\sum _{j=1}^{+\mathrm{\infty }}\sum _{k=1}^{+\mathrm{\infty }}\left\{j^2{\int }_0^{+\mathrm{\infty }}x{\mathrm{e}}^{-2j^2{x}^2-2k^2(y-x{)}^2}\mathrm{d}x\right\}\\ & =1-16\sum _{j=1}^{+\mathrm{\infty }}\sum _{k=1}^{+\mathrm{\infty }}\left\{j^2{\mathrm{e}}^{-\frac{2k^2{j}^2}{k^2+j^2}{y}^2}{\int }_0^{+\mathrm{\infty }}x{\mathrm{e}}^{-2(k^2+j^2)(x-\frac{k^2}{k^2+j^2}y{)}^2}\mathrm{d}x\right\}.\end{array}$$

Let

$$\begin{array}{rl}a& ={\mathrm{e}}^{-\frac{2k^2{j}^2}{k^2+j^2}{y}^2},\\ I& =a{\int }_0^{+\mathrm{\infty }}x{\mathrm{e}}^{-2(k^2+j^2){\left(x-\frac{k^2}{k^2+j^2}y\right)}^2}\mathrm{d}x,\end{array}$$

then we have

$$\begin{array}{rl}I& =a{\int }_0^{+\mathrm{\infty }}\left(x-\frac{k^2}{k^2+j^2}y\right){\mathrm{e}}^{-2(k^2+j^2){\left(x-\frac{k^2}{k^2+j^2}y\right)}^2}\mathrm{d}x\\ & +a{\int }_0^{+\mathrm{\infty }}\frac{k^2}{k^2+j^2}y{\mathrm{e}}^{-2(k^2+j^2){\left(x-\frac{k^2}{k^2+j^2}y\right)}^2}\mathrm{d}x\\ & =\frac{a}{4(k^2+j^2)}{\mathrm{e}}^{-2(k^2+j^2)\frac{k^4{y}^2}{(k^2+j^2{)}^2}}+\frac{a{k}^{2}y}{k^2+j^2}\frac{\sqrt{2\pi }}{2\sqrt{k^2+j^2}}\mathrm{\Phi }\left(\frac{2k^{2}y}{\sqrt{k^2+j^2}}\right)\\ & =\frac{1}{4(k^2+j^2)}{\mathrm{e}}^{-2k^2{y}^2}+\frac{k^2\sqrt{\pi }y}{\sqrt{2}(k^2+j^2{)}^{3/2}}{\mathrm{e}}^{-\frac{2k^2{j}^2}{k^2+j^2}{y}^2}\mathrm{\Phi }\left(\frac{2k^{2}y}{\sqrt{k^2+j^2}}\right),\end{array}$$

where $\mathrm{\Phi }\left(x\right)$ is the standard normal distribution function. Hence, it yields that

$$\begin{array}{rl}& P(X_1+X_2\le y)\\ & =1-16\sum _{j=1}^{+\mathrm{\infty }}\sum _{k=1}^{+\mathrm{\infty }}\left\{\frac{j^2}{4(k^2+j^2)}{\mathrm{e}}^{-2k^2{y}^2}+\frac{j^2{k}^2\sqrt{\pi }y}{\sqrt{2}(k^2+j^2{)}^{3/2}}{\mathrm{e}}^{-\frac{2k^2{j}^2}{k^2+j^2}{y}^2}\mathrm{\Phi }\left(\frac{2k^{2}y}{\sqrt{k^2+j^2}}\right)\right\},\end{array}$$

that is,

$$K\ast K\left(z\right)=1-16\sum _{j=1}^{+\mathrm{\infty }}\sum _{k=1}^{+\mathrm{\infty }}\left\{\frac{j^2}{4(k^2+j^2)}{\mathrm{e}}^{-2k^2{z}^2}+\frac{j^2{k}^2\sqrt{\pi }z}{\sqrt{2}(k^2+j^2{)}^{3/2}}{\mathrm{e}}^{-\frac{2k^2{j}^2}{k^2+j^2}{z}^2}\mathrm{\Phi }\left(\frac{2k^{2}z}{\sqrt{k^2+j^2}}\right)\right\}.$$

Appendix 4. Another aggregation scheme for test decision results

In the DCR-KS test procedure, the $(1-\alpha )$th quantile of the asymptotic distribution of the test statistic $U_{m,n}^a$ is used as the critical value in the aggregation step which implies that m should not be too small. In this case, an alternative aggregation step for the DCR-KS test, called DCR*-KS, can be adopted. Specifically, we still consider the following hypothesis testing in the aggregation step

$$H_0^a:{\alpha }_n\le \alpha ,versus{H}_1^a:{\alpha }_n>\alpha .$$ (A1)

The constructed test statistic is

$$U_{m,n}^{a\ast }=\sum _{i=1}^m{h}_n^{\left(i\right)},$$

which is the likelihood ratio test statistic for (D1), Then, the DCR*-KS testing scheme is

$$Reject{H}_{0}if{U}_{m,n}^{a\ast }>T_{1-\alpha }^m,$$

where $T_{1-\alpha }^m$ is the $(1-\alpha )$th quantile of the Binomial distribution $B(m,\alpha )$.

Disclosure statement

No potential conflict of interest was reported by the author(s).