Estimating a graphical intra-class correlation coefficient (GICC) using multivariate probit-linear mixed models

Abstract

Data reproducibility is a critical issue in all scientific experiments. In this manuscript, the problem of quantifying the reproducibility of graphical measurements is considered. The image intra-class correlation coefficient (I2C2) is generalized and the graphical intra-class correlation coefficient (GICC) is proposed for such purpose. The concept for GICC is based on multivariate probit-linear mixed effect models. A Markov Chain Monte Carlo EM (mcm-cEM) algorithm is used for estimating the GICC. Simulation results with varied settings are demonstrated and our method is applied to the KIRBY21 test-retest dataset.

1. Introduction

A crucial question in any statistical analysis is: how reliable is the data? Experimental replication for the purpose of measuring the reliability of measurements is the most common method for establishing reproducibility. In this paper, we consider repeated measurement of graphs and propose the concept of the graphical intra-class correlation coefficient for measuring their reliability.

The Intra-class correlation coefficient (ICC) has been proposed [1] and used to evaluate the reliability of measurements in a variety applications [2], [3]. ANOVA mixed-effect models have been proposed [4] as a framework for estimating the ICC. Suppose y_ij denotes the j^th measurement of subject i, x_i denotes the subject specific random effect and u_ij indicates the measurement error. The one-way ANOVA model is:

$$y_{ij}=\mu +x_i+u_{ij}{x}_i~N(0,{\sigma }_x^2),u_{ij}~N(0,{\sigma }_u^2),i.i.d.$$ (1)

The ICC is then defined as:

$$ICC=\frac{{\sigma }_x^2}{{\sigma }_x^2+{\sigma }_u^2}.$$ (2)

In (1) and (2), the total variability of the data is decomposed into subject-specific variability and measurement error; ICC represents the proportion of variability that is due to heterogeneity in subjects. In recent research, the ICC has been generalized to multivariate cases. The work in [5] proposed a model analogous to (1) in functional data using multilevel functional principal component analysis (MFPCA) and an image intra-class correlation coefficient (I2C2) was subsequently proposed in [6] to calculate ICC for image data.

Graphical data are becoming increasingly popular in scientific research. Notably, graphs are used in describing brain networks in neuroimaging. In such research, binary graphs are often obtained from functional magnetic resonance image (fMRI) [7], [8], [9], [10], [11]. The increasing number of graphical datasets motivates us to evaluate the reliability of binary graphs.

Figure. 1 illustrates idealized graphical measurements for three different subjects. Here each subject is measured three times. The left panel shows a case where graphical measurements resemble each other within one subject. The ICC, consequentially, should be higher. The right panel, on the other hand, demonstrates the opposite situation, where the repeated measurements within one subject show poor consistency. In such case, the ICC should be relatively lower. In this manuscript, we propose the concept of the graphical ICC (GICC) to quantify the similarity between repeated measurements of binary graphs. In Figure. 1, each binary graph is represented by a 0 – 1 vector. For example, the first graph of subject 1 is represented by (1, 1, 0, 1, 0, 0)^T1. Thus our goal is to define an ICC for multivariate binary data.

Figure 1.

The left panel shows a high GICC case, where graphical measurements are similar within subjects. The right panel illustrates a low GICC case, where graphical measurements are less consistent within subjects.

Many authors have discussed the ICC for single variate binary data. [12] proposed a moment based estimator. Probit linear mixed-effect models were used by [13] and [14] to estimate a confidence interval for binary data ICC.

There is also work discussing the similarity between graphs. The work in [15] and [16] discussed the similarity between nodes and edges in graphs. One main purpose of these papers was to find assembled subgraphs between two graphs. Instead of having a fixed node-to-node or edge-to-edge match, they found the match between two graphs based on edge/node similarity score.

Our objective, on the other hand, is to estimate the ICC to evaluate the reliability of replicated measurement of binary graphs. In section 2, a multivariate probit linear mixed model is proposed. A Monte Carlo expectation maximization (MCEM) algorithm will be discussed in section 3. Simulation results with various settings will be shown in section 4 and the results of our method being implemented on binary brain connectivity maps are in section 5. We will summarize the paper in section 6.

2. Model

Suppose {o_ij(d) : i = 1, …, I; j = 1 …, J_i; d = 1, …, D, } are binary observations representing repeated graph measurements for multiple subjects. Here, I is the total number of subjects, J_i is the number of visits for the i^th subject and D is the number of possible edges for all graphs. Usually, we have $D=\frac{N(N-1)}{2}$ where N is the number of nodes. In Figure. 1, for example, we have I = 3, J_i = 3, N = 4, D = 6. The multivariate probit-linear mixed model is as follows:

$${\mathrm{\Phi }}^{-1}(P(o_{ij}(d)|x_i(d)))=\mu (d)+x_i(d),{\mathbf{x}}_i~\mathbf{N}(\mathbf{0},{\mathbf{\Sigma }}_x),$$ (3)

where x_i = (x_i(1), …, x_i(D))^T. The GICC, is then defined as:

$$GICC=\frac{\mathit{\text{tr}}({\mathbf{\Sigma }}_x)}{\mathit{\text{tr}}({\mathbf{\Sigma }}_x)+D}.$$ (4)

For the purpose of estimation, the model can also be viewed as a threshold model that dichotomizes the observations from a latent Gaussian distribution. In other words:

$$o_{ij}(d)={\mathbf{I}}_{(y_{ij}(d)>0),}{y}_{ij}(d)=\mu (d)+x_i(d)+u_{ij}(d),{\mathbf{x}}_i~\mathbf{N}(\mathbf{0},{\mathbf{\Sigma }}_x),i.i.d.,{\mathbf{u}}_{ij}~\mathbf{N}(\mathbf{0},\mathbf{I}),i.i.d.,$$ (5)

where x_i = (x_i(1), …, x_i(d))^T and u_ij = (u_ij(1), …, u_ij(d))^T. The equivalency of these two models can be easily shown by the following calculation:

$$P(o_{ij}(d)=1|x_i(d))=P(y_{ij}(d)>0|x_i(d))=P(u_{ij}(d)>-(\mu (d)+x_i(d))|x_i(d))=1-\mathrm{\Phi }(-\mu (d)+x_i(d)))=\mathrm{\Phi }(\mu (d)+x_i(d)).$$

Formula 4 is a direct generalization from the univariate ICC Formula 2. GICC = 0 indicates that tr(Σ_x) = 0, which means that the between group variance is zero for all dimensions, d = 1, …, D. GICC ≈ 1 indicates that tr(Σ_x) ≫ D, implying that the variation between subjects is much larger than the variation within subject. When GICC = 0.5, tr(Σ_x) = D, implying that the overall between subject variation is equal to the within subject variation.

The advantage of using the trace is that: (1) it is an overall statistic instead of a edge specific statistic, which provides a global measurement to quantify graphical reproducibility; (2) compared to other numerical methods (e.g. max(diag(Σ)), ∑ (σ_ij)), the trace is invariant to orthogonal transformations, which is critical in measuring the variability of vectors.

Furthermore, using trace for measuring the global reproducibility was also proposed for the image ICC (I2C2) (see [6]) in and the functional version of ICC (see [5]).

3. The Monte Carlo EM Algorithm

MCEM algorithms have been used in probit-linear mixed models with single variate outcomes [17]. Here MCEM is generalized to the multivariate case. In model 5, the parameters of interest are μ and Σ_x. In the procedure of estimation, we treat o as observed data and [y, x] as the full data.

3.1. M-step

Given the full data y and x, the MLE for both parameters yields an explicit form:

$$\hat{\mu }=\frac{1}{{\sum }_i{J}_i}\sum _i\sum _j({\mathbf{y}}_{ij}-{\mathbf{x}}_i),{\hat{\mathbf{\Sigma }}}_x=\frac{1}{I}\sum _i{\mathbf{x}}_i{\mathbf{x}}_i^T.$$ (6)

Unlike [18], the estimate of μ does not involve Σ_x, since x is also treated as part of the complete data. So μ̂ is obtained based on both x and y, rather than only on y.

Substituting y, x and xx^T with E[y|o], E[x|o] and E[xx^T |o] respectively on the right side of 6, we obtain the M-step.

3.2. E-step

Based on 6, it is necessary to calculate E(y_ij |o), E(x_i|o) and $E({\mathbf{x}}_i{\mathbf{x}}_i^T|\mathbf{o})$. Note that

$$E[{\mathbf{x}}_i|\mathbf{o}]=E[E[{\mathbf{x}}_i|\mathbf{y}]|\mathbf{o}],E[{\mathbf{x}}_i{\mathbf{x}}_i^T|\mathbf{o}]=E[E[{\mathbf{x}}_i{\mathbf{x}}_i^T|\mathbf{y}]|\mathbf{o}].$$ (7)

The inner expectation can be obtained by using the joint distribution of {x_i, y_i1, …, y_iJi}. Noticing the following fact:

$$[{\mathbf{x}}_i,{\mathbf{y}}_{i1},\dots ,{\mathbf{y}}_{i{J}_i}]=\prod _j[{\mathbf{y}}_{ij}|{\mathbf{x}}_i]\times [{\mathbf{x}}_i]\propto \text{exp}\{-\frac{1}{2}[\sum _j\{{({\mathbf{y}}_{ij}-\mu -{\mathbf{x}}_i)}^T({\mathbf{y}}_{ij}-\mu -{\mathbf{x}}_i)\}+{\mathbf{x}}_i^T{\mathbf{\Sigma }}_x^{-1}{\mathbf{x}}_i]\}\propto \text{exp}\{-\frac{1}{2}[{\mathbf{x}}_i^T(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1}){\mathbf{x}}_i-2{[\sum _j({\mathbf{y}}_{ij}-\mu )]}^T{\mathbf{x}}_i]\},$$ (8)

it can be derived that:

$${\mathbf{x}}_i|{\mathbf{y}}_{i1},\dots {\mathbf{y}}_{i{J}_i}~\mathbf{N}({(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}({\mathbf{y}}_{i․}-J_i\mu ),{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}),$$ (9)

where y_i․ = ∑_j y_ij. Thus we have

$$E({\mathbf{x}}_i|\mathbf{O})=E[E[{\mathbf{x}}_i|\mathbf{y}]|\mathbf{o}]=E[{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}({\mathbf{y}}_{i․}-J_i\mu )|\mathbf{o}]={(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}(E[{\mathbf{y}}_{i․}|O]-J_i\mu ),$$

$$E[{\mathbf{x}}_i{\mathbf{x}}_i^T|\mathbf{o}]=E[E[{\mathbf{x}}_i{\mathbf{x}}_i^T|\mathbf{y}]|\mathbf{o}]=E[{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}({\mathbf{y}}_{i․}-J_i\mu ){({\mathbf{y}}_{i․}-J_i\mu )}^T{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}+{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}|\mathbf{o}]={(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}E[({\mathbf{y}}_{i․}-J_i\mu ){({\mathbf{y}}_{i․}-J_i\mu )}^T|\mathbf{o}]{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}+{(J_i\mathbf{I}+{\mathbf{\Sigma }}_x^{-1})}^{-1}.$$ (10)

However, the term E[y_i․|o] and $E[{\mathbf{y}}_{i․}^T{\mathbf{y}}_{i․}|\mathbf{o}]$ does not have an explicit form. Here we use a Gibbs sampler to approximate the conditional expectation. Notice that, given o, the distribution of y is multivariate truncated normal. The Gibbs sampler for such a distribution has been discussed in [19], [20], [21]. In the Gibbs sampling cycles, we choose the burn in period to be the first T = 200 and treat the following B = 500 elements as limiting realizations from the conditional distribution of y|o. Then an empirical conditional expectation is calculated as follows:

$$Ê[{\mathbf{y}}_{ij}|\mathbf{o}]=\frac{1}{B}\sum _{b=T+1}^{T+B}{\mathbf{y}}_{ij}^{(b)},$$ (11)

$$Ê[{\mathbf{y}}_{i․}{\mathbf{y}}_{i․}^T|\mathbf{o}]=\frac{1}{B}\sum _{b=T+1}^{T+B}{\mathbf{y}}_{i․}^{(b)}{\mathbf{y}}_{i․}^{(b)T}.$$

3.3. Observed information matrix for μ

Though we are not specifically interested in estimating μ for the graphical ICC, the estimate of μ with its standard error remains of potential interests, especially for modeling multivariate binary data using probit-linear mixed model. [22] expressed the observed information matrix in EM algorithm using the first and second derivative of the full likelihood.

Assume the observed log-likelihood is l_o(o, θ) where θ = (μ, Σ_x) and the full log-likelihood is l_x,y(x, y, θ), following [22], we have:

$$I_{\mathbf{o}}(\theta )=E_{\theta }[-\frac{{\partial }^2{l}_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu \partial {\mu }^T}|\mathbf{o}]-E_{\theta }[\left(\frac{\partial l_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu }\right){\left(\frac{\partial l_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu }\right)}^T\left|\mathbf{o}\right]+\left(\frac{\partial l_o(\mathbf{o},\theta )}{\partial \mu }\right){\left(\frac{\partial l_{\mathbf{o}}(\mathbf{o},\theta )}{\partial \mu }\right)}^T.$$ (12)

Let I_o = I_o(θ̂), where θ̂ is the maximum likelihood estimator. Then we have:

$$I_{\mathbf{o}}(\hat{\theta })=E_{\hat{\theta }}[-\frac{{\partial }^2{l}_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu \partial {\mu }^T}|\mathbf{o}]-E_{\hat{\theta }}[\left(\frac{\partial l_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu }\right){\left(\frac{\partial l_{\mathbf{x},\mathbf{y}}(\mathbf{x},\mathbf{y},\theta )}{\partial \mu }\right)}^T\left|\mathbf{o}\right].$$ (13)

Following the same path as in the E-step, we can use Gibbs sampler and empirical averages to approximate the conditional expectation.

4. Simulation

4.1. Estimates

We set number of subjects at I = 100, 200 and each subject receives J = 2, 4 repeated measurements. The number of nodes is set to be N = 5 so that the number of possible undirected edges is D = 10. The true μ is set to be 0.5 for all elements and

$${\mathrm{\Sigma }}_x[i,j]=r{\rho }^{|i-j|},\text{where}\rho =0.8.$$

The underlying true graphical ICC using definition 4 is controlled by r. We set r = 2, 4 in each setting so that the corresponding ICC’s are $\frac{rD}{rD+D}=2/3$ and 4/5 respectively. A total of 500 simulations were run in each simulation group.

In Table. 1, the average estimated GICC for r = 2 groups are 0.702, 0.672, 0.683 for I₁₀₀J₂, I₁₀₀J₄ and I₂₀₀J₄ group respectively, comparing to an under-lying truth 2/3 ≈ 0.667. As number of individuals increases, or as the number of repeated measurements increases, both the bias and the standard deviation of the estimated GICC reduces. When r = 4, the average estimated graphical ICCs are 0.817, 0.800 and 0.806, respectively. The MLE of GICC in each case has a positive bias, which is reduced as either I or J_i increases.

Table 1.

Simulation results

Setting	Estimates for σii					ICC est.
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 100, J = 2, r = 2	2.37 (1.18)	2.48 (1.09)	2.38 (1.11)	2.35 (1.04)	2.34 (1.01)	0.702
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 2/3	2.45 (1.09)	2.36 (1.06)	2.43 (1.13)	2.43 (1.04)	2.38 (1.18)	(0.033)
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 100, J = 2, r = 4	4.56 (2.11)	4.74 (2.20)	4.62 (2.30)	4.40 (2.01)	4.56 (2.28)	0.817
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 4/5	4.65 (2.12)	4.63 (2.04)	4.52 (2.07)	4.61 (2.04)	4.51 (2.31)	(0.025)
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 100, J = 4, r = 2	2.02 (0.60)	2.10 (0.64)	2.07 (0.61)	2.04 (0.65)	2.10 (0.65)	0.672
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 2/3	2.08 (0.59)	2.06 (0.60)	2.08 (0.63)	2.08 (0.65)	2.05 (0.61)	(0.026)
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 100, J = 4, r = 4	4.00 (1.29)	4.08 (1.36)	4.04 (1.28)	3.96 (1.29)	4.17 (1.36)	0.800
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 4/5	4.04 (1.24)	4.04 (1.20)	4.10 (1.34)	4.09 (1.32)	4.05 (1.24)	(0.020)
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 200, J = 2, r = 2	2.08 (0.68)	2.22 (0.77)	2.23 (0.74)	2.17 (0.73)	2.19 (0.70)	0.683
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 2/3	2.17 (0.76)	2.17 (0.68)	2.16 (0.71)	2.20 (0.77)	2.13 (0.74)	(0.026)
	σ1,1	σ2,2	σ3,3	σ4,4	σ5,5	ICC
I = 200, J = 2, r = 4	4.07 (1.38)	4.33 (1.41)	4.25 (1.57)	4.20 (1.49)	4.18 (1.42)	0.806
	σ6,6	σ7,7	σ8,8	σ9,9	σ10,10
ICCtrue = 4/5	4.20 (1.52)	4.20 (1.36)	4.25 (1.43)	4.30 (1.47)	4.15 (1.47)	(0.020)

4.2. Robustness

In the proposed multivariate probit-linear mixed model, the assumption was made that the underlying within group error term is independent across edges. In other words, in model 5, the off-diagonal elements of var(u_ij) is set to be zero. In real applications, this may not be the case. Therefore, we conducted simulations using the above settings when I = 100, J = 2 and r = 2. However, instead of setting Σ = I, we set σ_ij = ρ^|i−j|, where ρ = 0.1 and 0.5. The resulting GICC based on 100 simulations have means 0.699(0.025) and 0.722(0.026). The results highlight that the resulting GICC is still close to the underlying truth when there are correlations between edges in the error term.

4.3. Comparison With Other Benchmarks

We compared our proposed method with other available methods using our first simulation case (I = 100; J = 2; r = 2). The work in [23], [24] and [25] used ICC derived from one-way ANOVA, [13] proposed ICC for binary data using a single variate probit model. It should be pointed out that, all the other methods are based on the single variate ICC, so the comparison is limited. We compare our method with (1) average ICC(1)’s for all edges based on one-way ANOVA, (2) ICC(1) for the mean of the binary vector and (3) average edge-wise ICC based on single variate probit model. The results are shown in Table 2.

Table 2.

Compare to Benchmark

	Models
	ave. ICC(1)	ICC(1) for vec.mean	1-var probit	GICC
ICCtrue = 0.667	0.457 (0.028)	0.812 (0.035)	0.613 (0.025)	0.696 (0.029)

First of all, the first two ICCs are all derived from the one-way ANOVA model, thus all of the binary data are considered to be continuous. The average edge-wise ICC is only 0.457, but ICC for the mean vector is 0.812. None of them are close to the truth. The edge-wise ICC treats all binary data as pure jumps instead of treating them as having underlying continuous data. The ICC for the mean of vector can only provide the ICC on the average statistic, but can not align the data with the same edge. The average edge-wise binary ICC based on single variate probit model also yields lower ICC. It also shows the difference between an unstructured covariance matrix for x and a diagonal matrix.

4.4. Running Time

To evaluate the running time of the MCEM algorithm, we conducted simulations with varied settings in which the number of subjects = 50, 100, the number of replicates = 2, 3, 4 and the number of edges = 10, 20 and 30. To be consistent with all settings, we terminated the algorithm after 30 iterations. All simulations were conducted using a 2.4GHz core on PowerEdge C6145 AMD Processor-based 2U Rack Server.

Table 3 shows that the number of subjects has a strict linear relationship with running time. Running time increases nonlinearly with either the number of replicates or the number of edges in the graph. With under 30-edge graphs and 4 replicates, the running time is less than twenty minutes. It is clear that if the number of replicates is 4, the running time grows faster than a quadratic function of the number of edges. Therefore, the current algorithm requires a relative small number of edges for each graph. We will further discuss it in Section 6.

Table 3.

Running Time (in seconds)

	I=50			I=100
	D = 10	D = 20	D = 30	D = 10	D = 20	D = 30
J = 2	17.2	28.7	62.4	33.5	57.2	124.8
J = 3	20.7	61.5	215.4	41.3	122.8	430.6
J = 4	28.0	144.5	586.5	55.9	289.0	1173.3

5. Application

Resting-state fMRI scans consisted of a test-retest dataset previously acquired at the FM Kirby Research Center at the Kennedy Krieger Institute, Johns Hopkins University [26] are used to highlight the method. Twenty one healthy volunteers with no history of neurological disease each underwent two separate resting state fMRI sessions on the same scanner. A 3T MR scanner was used (Achieva, Philips Healthcare, Best, The Netherlands) utilizing a body coil with a 2D echoplanar (EPI) sequence and eight channel phased array SENSitivity Encoding (SENSE; factor of 2) with the following parameters: TR 2s, 3mm × 3mm in plane resolution, slice gap 1mm, for total imaging time of 7 minutes and 14 seconds. One subject was excluded due to technical issues at acquisition.

ICA (Independent Component Analysis) was performed using MEDOLIC (Multivariate Exploratory Linear Optimized Decomposition into Independent Components) version 3.10 in FSL (FMRIB Software Library, FMRIB, Oxford, UK). Preprocessing included removal of low-frequency drift with a highpass filter cutoff of 250s, realignment of the fMRI time series using MCFLIRT, slice timing correction, brain extraction using BET, and spatial smoothing with FWHM of 6mm. Images were registered to MNI standard space with resampling resolution of 2mm. ICA was performed using multi-session temporal concatenation with automatic dimensionality estimation and time-course variance normalization implemented in MELODIC. 43 components were identified by MELODIC.

Relevant ICA components corresponding to known large scale brain networks were identified by a board certified neuroradiologist with experience in resting state fMRI. Seven total components were selected (default mode network, dorsal attention network, motor network, visual network, salience network, and two lateralized executive control networks), and the 7 by 7 correlation matrix was calculated (see raw data in Figure 5). Different thresholds were used to dichotomize the raw graphs into binary ones, where the thresholds were chosen from 0.1 to 0.8 using grid 0.01 (see Figure 5). The GICC algorithm was then implemented on these binary graphs.

The GICC was then calculated for each threshold (see Figure 3). The GICC remains above 0.6 when the threshold is between 0.1 and 0.6. For threshold outside of this band the GICC decreases dramatically. When the threshold is around 0.8, GICC fluctuates more significantly and the value eventually drops to 0.1. Thus the GICC shows high reproducibility of the raw data if a reasonable threshold is employed (from 0.1 to 0.6). When the threshold is too high, only few raw values will be dichotomize to 1, such that poor reproducibility is obtained. For practical subsequent applications, one could use the value that maximizes the GICC in this data set (see Figure 5).

Figure 3.

The calculated GICC under different thresholds. The threshold were picked equally spaced from 0.1 to 0.8 using grid 0.01. The maximized GICC is indicated in the figure, which corresponds to a 0.35 threshold.

6. Conclusion

In this paper, we propose the concept of the graphical intra-class correlation coefficient using multivariate probit mixed-linear models. The GICC is defined as $\frac{tr({\mathrm{\Sigma }}_x)}{tr({\mathrm{\Sigma }}_x)+D}$. We used a Monte Carlo EM algorithm to obtain the MLE of Σ_x, while a Gibbs sampler was used in the E-step. We show the results of GICC in varied simulation settings and in the KIRBY21 test-retest datasets.

While providing GICC, the estimation procedure can also be generalized to multivariate probit mixed-linear model with fixed and random covariates components, which is:

$$o_{ij}(d)={\mathbf{I}}_{(y_{ij}(d)>0)}{y}_{ij}(d)=\sum _{p=1}^P{\mu }_{ip}(d){\beta }_p(d)+\sum _{r=1}^R{x}_{ir}(d){\eta }_{ir}(d)+u_{ij}(d){\eta }_{ir}~\mathbf{N}(\mathbf{0},{\mathbf{\Sigma }}_r),i.i.d.{\mathbf{u}}_{ij}~\mathbf{N}(\mathbf{0},\mathbf{I}),i.i.d‥$$

In the EM algorithm, η and y can be treated as full data and the procedure in Section 3 follows. In section 3.3, we also calculate the observed information matrix for the fixed effects which can provide confidence intervals for β’s. Moreover, the procedure can also be used multivariate generalized mixed-linear models, such as multivariate poisson or logistic regression.

Currently, our method works for small graphs. As the number of nodes in a graph increases, the number of parameters of interests grows quadratically (D ~ O(N²) and #{σ_ij} ~ O(D²)). Thus a graph with 100 nodes will have tens of millions of parameters to estimate. The Gibbs sampler could not be implemented effectively in such cases. Therefore, the algorithm currently requires a relatively small number of nodes for each graph (typically less than 10). In order to achieve faster convergence rate as well as control the Monte Carlo error induced by Gibbs sampler, ascent based MCEM [27] and acceleration EM algorithm ([28]) could be implemented.

Notice that from the application, GICC could also be used for choosing thresholds for dichotomizing raw graphs. The value that maximizes the GICC is a reasonable threshold, since it yields the best reproducibility of a well known benchmark data set.

In summary, GICC provides us a way to measure the reproducibility of repeated graphical measurements. The current algorithm gives us the estimates of GICC for relatively small graphs. To our knowledge, GICC for large graphs has not been addressed before and therefore deserves further investigation.

Figure 2.

The figure illustrates two repeated measurements for one subject. On the left, raw correlations between seven nodes are illustrated. Then the raw correlations are dichotomizing using different thresholds (0.2, 0.35, 0.6 are listed here). Our algorithm is then implemented on binary graphes using each threshold. Red suggests lower value and white (yellow) suggests higher value. In the binary graph on the right, red indicates 1 and yellow indicates 0.