Statistical Analysis of Repeated MicroRNA High-Throughput Data with Application to Human Heart Failure: A Review of Methodology

Abstract

Complex experimental designs present unique challenges in the analysis of microRNA (miRNA) Cycle to Threshold (Ct) values. In this manuscript, we discuss various statistical techniques and their application in an analysis performed at the JG Brown Cancer Center. We consider data quality evaluation, data normalization, and statistical hypothesis procedures all in context of the example. The experiment utilized as the motivating example involved repeated sampling over time. The intra-subject correlation created by the repeated sampling should be incorporated into the analysis resulting in additional significant miRNAs. The statistical techniques leveraged to analyze miRNA Ct values resulting from qPCR should incorporate key features of the experimental design. It discusses potential issues with the commonly used methodologies when the experiment collects multiple samples from the same individuals over time.

Introduction

Several studies have examined the role of miRNAs in various diseases such as cancer¹ and heart disease.² miRNAs are short, noncoding RNA molecules that affect gene expression. The clinical understanding of the role of miRNAs in disease is growing very quickly. There are several different techniques used to obtain the expression levels of the miRNAs and include microarray analysis and TaqMan PCR from Applied Biosystems.³ The reproducibility of experiments performed with Taq-Man PCR has also been investigated and found to be high.⁴ There are also several different normalization techniques that can be employed to remove systematic differences between samples that do not represent true biologic differences.⁵ The Ct value represents the cycle number at which the fluorescent signal of the reporter dye crosses a threshold value.⁶ The threshold is placed such that the PCR is in the exponential phase.

Typically, a hypothesis testing procedure is applied once the Ct values are normalized. The student’s t-test appears to be a popular procedure applied to compare the mean of the normalized Ct values between the two groups.^7–9 Then the p-values need to be adjusted to control the type I error rate using an appropriate method such as the Benjamin-Hochberg¹⁰ method.

The experiments are becoming more complex as they are designed to examine the relationships between the disease process, treatments, and the expression of miRNAs. They often involve repeated measurements on the same subjects over time and require specialized statistical techniques to handle the additional correlation. Montenegro et al.¹¹ developed an experiment that examines the expression of miRNAs at different gestational ages. The authors used a Generalized Estimating Equation (GEE)¹² model

$$g\left(E\right[Y_{ijk}\mid x_{ij}\left]\right)$$ (1)

with an exchangeable correlation structure where Y_ijk is the k^th Ct value for the i^th subject and the j^th gestational age. The x_ij is the j^th covariate for the i^th subject. The model included the obstetric condition and gestational age. The GEE model is in the class of semi-parametric models since it does not require full specification of the likelihood to calculate the parameter estimates. The model is easily applied in the repeated sampling situation created by the qPCR experiment.

The experimental designs present unique challenges to the analyst. The motivation of this manuscript is an analysis of miRNA Ct values performed at the James Graham Brown Cancer Center. The experiment was designed to perform an exploratory analysis of changes in the cardiac expression of miRNAs in patients with end-stage heart failure (HF) undergoing placement of a left ventricular assist device (LVAD) and subsequent heart transplantation. The experimental design presented some unique challenges in the analysis of the data that require a description of the experiment for full appreciation. The remaining sections will describe the experiment, compare various analysis techniques, finally discuss the results, and provide conclusions.

Motivating Example

The experiment is designed to analyze miRNA expression profiles in patients with advanced HF undergoing surgical implantation of an LVAD (a mechanical pump designed to assist in blood flow from the weakened heart) as a bridge to heart transplantation, i.e., to maintain the patient until the heart can be replaced.

The initial assessment of miRNA expression levels is an exploratory analysis of 384 unique miRNAs. Number of the selected miRNAs are carefully selected based on resources and their known features in heart functioning at the time of experimentation. Each subject has a sample of the left ventricle removed at the time of LVAD implantation (IMP), a sample taken of the left ventricle at the time of heart transplant and LVAD explantation (ELV), and a sample taken of the right ventricle at the time of LVAD explant (ERV). Therefore, each person receiving an LVAD in the study contributed three samples at two different time periods. The term subject refers to the patient under study. The plate refers to one of the three biologic samples that the subject contributed. The plate is the specific set of miRNAs contributed by a subject from a specific point in time and location in the heart. All participants signed an informed consent form for the use of the tissue and the study was approved by the University of Louisville’s Institutional Review Board (IRB, IRB# 101.04JH). There are also archived control samples which represent hearts not experiencing failure.

The experiment is intended to be an exploratory analysis and there are a limited number of wells. Therefore, there are no technical replicates in order to maximize the number of the miRNAs to be included in the analysis. Each of the 384 wells contains a unique miRNA except for the endogenous controls which may be repeated a few times. There are challenges created by the time required to collect the data as well as the multiple time points in the trial. The specific challenges will be discussed in the following section.

Statistical Methodologies

The experimental design presents two unique challenges to the analyst. First, the experiment did not include technical replicates in order to maximum the number of unique miRNAs to include in the experiment. This implies the normal data quality techniques are not available but a different approach is required. Second, the repeated sampling of miRNAs from the same subjects over time presents a challenge to the analyst. The typical normalization techniques are not designed to preserve the naturally occurring correlation structures. There are statistical models that can include the correlation structure which can be employed.

In this manuscript, we discuss the most commonly used methodologies or methodologies with readily available software. We apply them to the data discussed in the motivating example.

Data Quality

The data quality had to be assessed once the data was ready for analysis. Usually, the technical replicates are used to assess the quality of the Ct values. The technical replicates can be used to determine if the information is truly missing or missing at random. We defined missing to be a Ct value greater than 35 even though the software can detect values up to 40. If the values are missing at random, then an imputation algorithm can be utilized, while truly missing values should be unaltered. We had to approach the problem differently since the experiment was constructed to include as many unique miRNAs as possible excluding technical replicates.

The general approach to the specific problem was completed in two steps. First, we calculated the number of plates with values larger than 35 for each miRNA. All of the plates regardless of time during treatment were simultaneously included in the analysis. Then miRNAs that were missing across a large number of plates were excluded from the study. The benefit is a reduction in the number of miRNAs included in the hypothesis testing.

Although using a fixed threshold value is suggested and routinely used, but is subject to selection bias. Another alternative is to use a varying threshold for each plate, in which case we need to combine CT values and varying threshold values in a parametric model that we consider in another manuscript.

Normalization

The second issue encountered during the analysis of the Ct values was appropriate normalization. Normalization is required to remove unwanted technical variation present in the sample.¹³ Many of the normalization techniques are from the analyses of mircroarray datasets and may not be completely applicable. The number of measurements is much smaller in the miRNA data and the majority of miRNA are not expressed or expressed at very low levels.⁵

In this analysis, we considered the delta-Ct method,¹⁴ the mean normalization,¹³ quantile normalization,15 and rank invariant normalization.¹⁶ The normalization techniques represent commonly used techniques with readily available software. The coefficient of variation associated with the raw data is included as a reference to evaluate the normalization techniques against.

Let N be the total number of subjects included in the study after filtering and i = 1,…, N be the individual patient number. Let j = 1, 2, 3 be the repeated sample number for each subject. In the motivating example, j = 1 corresponds with the IMP biologic sample, j = 2 corresponds with the ELV biologic sample, and j = 3 corresponds with the biologic ERV sample. Then M = 3N is the total number of plates included in the experiment where m = 1,…,M. We will also let K be the unique count of miRNAs included in the analysis after filtering where k indicates the k^th miRNA for k = 1, 2, …, K. Then Ct_ijk represents the Ct value from the i^th subject, j^th sample, and k^th miRNA. For simplicity, the calculations that are plate specific will be discussed in terms of the subscript m where m = 1, …, M. Note that M = 3N which represents the total number of plates to be analyzed. The calculations that are sample and person specific will include all three subscripts i, j, and k.

Delta-Ct

The delta-Ct method subtracts the mean of the endogenous controls from the remaining Ct values. Two endogenous controls were selected for the analysis RNU24 and RNU48. The algebraic equation representing this is

$$\Delta C{t}_{\mathrm{mk}}=C{t}_{\mathrm{mk}}-C{t}_{\mathrm{e}}$$ (2)

where Ct_e is average of the Ct values from the endogenous controls and Ct_mk is the individual values for all the other miRNAs in the sample. The delta-Ct is a popular method of normalization due to the natural biologic motivation and explanation which is contained in the Appendix.

Mean Normalization

The mean normalization subtracts the average of plate m’s Ct values from all Ct values contained on plate m. The mathematical representation is

$$\Delta C{t}^m=C{t}_{mk}-\frac{\sum _{k=1}^{K}C{t}_{mk}}{K},k=1,\cdots ,Kandm=1,\cdots ,M$$ (3)

where Ct_mk is the k^th miRNA from the m^th plate and M = 3N . The method is similar, in essence, to the Delta-Ct method but relies on an average of all Ct values to perform the normalization.

Quantile Normalization

The quantile normalization forces the distribution of Ct values to be the same across all the plates. The method takes the largest value and replaces it with the mean of the largest values, and then repeats for each subsequent data point. Let

$$q_k^{\ast }=\frac{1}{\sqrt{M}}\sum _{m=1}^M{q}_{mk}d=\left(\frac{1}{M}\sum _{m=1}^M{q}_{mk},\cdots ,\frac{1}{M}\sum _{m=1}^M{q}_{mk}\right)$$ (4)

where

$$d=\left(\frac{1}{\sqrt{M}},\cdots ,\frac{1}{\sqrt{M}}\right)$$ (5)

and q_k is the k^th row of ordered Ct values. The Ct values are ordered for each plate independent of the other plates. The quantile normalization methodology is commonly used in the analysis of microarray expression values but the technique assumes that the distribution of the expression values is the same.

Rank Invariant Normalization

The rank invariant method attempts to determine miRNAs which whose rank does not change across the plates. The rank invariant miRNAs are then used to create a smooth curve applied to the entire sample. The rank invariant normalization is completed in two steps. First, the k^th miRNA is considered to be rank invariant if the absolute value of the change in the relative rank (r) of the miRNA in the $m_i^{th}$ plate and the $m_j^{th}$ plate is less than 0.05 or

$$\frac{\mid r_{m_{i}k}-r_{m_{j}k}\mid }{r_{m_{j}k}}<0.05.$$ (6)

Then a smooth line is fit through the set of rank invariant genes which is applied to all miRNAs. The rank invariant method is another technique resulting from the analysis of mircoarray expression values.

Coefficient of Variation

The cumulative distribution of the coefficient of variation is used to compare the various normalization techniques. We calculated the coefficient of variation for each miRNA over all the plates. Let K be the total number of miRNAs then the coefficient of variation is

$$C{V}_k=\frac{sd\left(Ct\right)}{mean\left(Ct\right)},k=1,2,\cdots ,K$$ (7)

Next we created a cumulative distribution of the coefficient of variation as

$$\tilde{F}=\frac{1}{K}\sum _{k=1}^{K}I\{C{V}_k\le t\}$$ (8)

Hypothesis Testing

The final issue to consider during the analysis was the appropriate hypothesis testing procedure. A t-test, Mann-Whitney U test, and a testing procedure proposed byPounds and Rai18 were considered as well as a model based on the GEE approach.

The experiment collected data on the same individuals at two different time points. The first sample was taken when the LVAD was implanted but only the left ventricle could be sampled. The second sample was taken when the heart transplant was performed and both the left and right ventricles were sampled.

GEE Model

A GEE model was constructed to consider the repeated sampling. The Y_ijk is the repeated Ct values taken at each of the time points described for each individual. The x_ij represent the covariates including the three groups which are the sample taken at the time of LVAD implant (IMP), the left ventricle sample at the time of explant (ELV), and the right ventricle sample at the time of explant (ERV). Contrasts were constructed to analyze the difference between the Ct values taken at the time of implant and the left ventricle sample at the time of explant as well as the difference in Ct values at the time of the explant between the left and right ventricles. The analysis currently assumes an exchangeable correlation structure which accounts for the correlation between the samples for each subject. The Gaussian distribution with the identity link function was selected given the relatively normal distribution of the normalized Ct.

t-test

If the intra-subject correlation is ignored then a paired t-test can be employed to compare the average expression values from two of the three samples. The mathematical notation is

$$t_d=\frac{\sqrt{N}\overline{d_k}}{s_d}$$ (9)

where

$${\stackrel{‒}{d}}_k=\frac{\sum _{n=1}^N(C{t}_{i1k}-C{t}_{i2k})}{N}$$ (10)

and Ct_i1k is the Ct value for the k^th miRNA from the first sample for the i^th subject. The value s_d is the appropriate estimate of the standard error for the paired difference. Under the null hypothesis of no difference, the test statistic, t_d, follows a t-distribution with N – 1 degrees of freedom.

Mann-Whitney U-test

The Mann-Whitney U test is a non-parameter test which evaluates the population medians based on two samples. The procedure also ignores the intra-subject correlation and does not require the assumption that sampling statistic follows the normal distribution. The Mann-Whitney U test statistic is

$$U=N^2+\frac{N(N+1)}{2}+R_1$$ (11)

where R₁ is the sum of the ranks, based on the entire combined sample, associated with just the first sample.

In both the t-test and the Mann-Whitney U-test, as well as the GEE model, there are a total K test statistics and corresponding hypothesis tests. A multiplicity adjustment should be applied in order to control the total type I error rate or the false discovery rate.

Assumption Adequacy Averaging

The concept of Assumption Adequacy Averaging (AAA) was proposed by Pounds and Rai¹⁸ as a technique for developing more robust methods that incorporate assessments of assumption adequacy into the analysis. The technique utilized empirical Bayesian principles described by Efron et a.,¹⁹ as well as Pounds and Morris,²⁰ to develop a method that averages the results from different testing procedures with weights determined by tests of assumption adequacy. The method combines results from the classical t-test and rank-sum tests with weights determined by the Shapiro-Wilk’s test to assess the normality assumption.

Sample Size Justification

Many of these studies are designed on ad-hock basis. Experimentations are not, usually, planned before, like in clinical trials. However, post analyses justification for sample size is essential. In high-throughput data analyses, where number of hypotheses exceeds excessively, one of the primary objective is to have a high probability of declaring a hypothesis (such as a miRNA) to be significant (differentially expressed) if they are truly significant (truly expressed), while keeping the probability of making false declarations low. There are two approaches for controlling the error rates: False Discovery Rate (FDR) and Family-Wise Error Rate (FWER). Following Benjamini and Hochberg (1995), the FDR is the expected value of the proportion of the non-prognostic genes (in our case miRNAs) among the discovered genes (in our case miRNAs). We will use FDR approach for sample size justification.

In our motivating example we have three repeat measurements. We plan to compare overall effect and two pairwise comparisons. The pairwise comparison is based on a paired t-test. It is better to use a two-sided test as we do not have prior information of a miRNA being down regulated or up-regulated, although this requires a slightly bigger sample size.

Adjusted significance level

Following Chow et al. (2008), the adjusted significance level is given as

$${\alpha }^{\ast }=\frac{r_{1}f}{m_0(1-f)}$$

In the above expression r₁ is the desired number of alternative hypothesis (# of miRNAs to be discovered) to be declared significant at f false discovery rate from m total hypotheses (total # of miRNAs), with m₁ potentially alternative hypotheses (potentially significant # of miRNAs) and m₀ (= m – m₁) null hypotheses (not significantly expressed miRNAs). Once level of significance is determined it is straight forward to determine design parameters (power, effect size or sample size).

Demonstration to Motivating example

Based on our experience and collaboration, we expect m1= 40 (around 10% of 384) of which we expect to identify about r1=10 miRNAs. The resulting adjusted significance level will be 0.0015 and 0.0032 at FDR=5% and 10%, respectively. Using a one sided, one-sided paired t test, with n=9 for 80% power, and significance level of 0.0015 and 0.0032, we can detect effect sizes of 1.69 SD (standard deviation) units and 1.52 SD, respectively. Assuming equal variances in the repeat measures, we can identify unit fold (ratio of means), a quantitiy most commonly used in the collaborative research, 2.69 fold for up-regulated or 0.37 fold down regulated at FDR level of 5%.

Results

The methods described above were applied to the Ct values resulting from the motivating example. The first item to consider is the data quality resulting from the qPCR experiment. One component of the analysis considered a comparison between the expression values from the left ventricle at the time of explant (ELV) to the expression values from the right ventricle at the time of explant (ERV). There were nine subjects with the PCR performed on the same platform. miRNAs with missing values on 13 or more of the 18 plates (or more than 72.2%) were excluded from further analysis.

A similar approach was used to reduce the number of Ct values greater than 35 in an analysis that considered the three samples (IMP, ELV, and ERV) in one model. miRNAs were excluded from future analysis if the Ct values were missing from 19 or more of the 27 plates (or more than 70.4%). After filtering, each individual contributes exactly the same miRNAs at each of the three time points resulting in balanced repeated design. Table 1 reports the percentage of the plates with Ct values larger than 35 for the different comparisons before and after filtering. In each case, the percentage of reasonable Ct values less than 35 increased.

Table 1.

Effect of Filtering on Percentage of Ct Values Deemed Undetermined

Experiment	Data	Before Filtering	After Filtering
ELV vs ERV	Ct values ≤ 35	61.1%	90.2%
	Ct values > 35	35.9%	9.8%
IMP vs ELV vs ERV	Ct values ≤ 35	69.5%	90.0%
	Ct values > 35	39.5%	10.0%

Notes: Abbreviations: – Cycle to Threshold, HF- end-stage heart failure, VAD - Left Ventricle Assist Device, GEE – Generalized Estimating Equations, IMP – the sample taken from the left ventricle at time of implant, ERV – the sample taken from the right ventricle at time of explant, ELV – the sample taken from the left ventricle at the time of the explant.

The four different normalization techniques were applied to the miRNA values after the filtering. The cumulative distribution associated with the various normalization techniques is depicted in Figure 1. The quantile normalization performs the best based on the reduction of the coefficient of variation since the graph goes to 1 at the fastest rate. The other methods introduce more variation than seen in the raw data. The endogenous controls were carefully selected for the delta-Ct method based on a review of literature including analysis conducted by Applied Biosystems.¹⁷ The stability of the controls was accessed as well and the average of the triplicates was used as the Ct_e.

Figure 1.

Cumulative Distribution of the Coefficient of Variation

The effects of the normalization techniques are depicted in Figures 2 and 3. Figure 2 displays the density estimates of the raw Ct values for each plate. We can see the distributions appear to follow a normal distribution with a small deviation near 40. The small deviation is caused by the number of miRNAs with Ct values of 40 which implies a Cycle to Threshold value was not determined. It also supports the choice of the quantile normalization since all of the curves are very similar in shape, location, and scale. Figure 3 displays the density estimates based on the quantile normalized Ct values. There are still 27 curves (or one for each plate) but the normalization technique forces each distribution to be nearly identical. The results appear to be extremely normal but the additional small deviation from the normal curve still appears near the Ct value 40.

Figure 2.

Density Estimate of Ct Values for Each Plate over all miRNAs - No Normalization

Figure 3.

Density Estimate of Ct Values for Each Plate over all miRNAs - Quantile Normalization

The final issue to consider during the analysis was the appropriate hypothesis testing procedure. A t-test, Mann-Whitney U test, and a testing procedure proposed by Pounds and Rai¹⁸ were considered as well as a model based on the GEE approach.

The distribution of the adjusted p-values is depicted in Figures 4 and 5. The False Discovery Rate described by Benjamini and Hochberg⁹ was used. We considered an adjusted p-value smaller than 5% as statistically significant. Figure 4 displays the distribution of the adjusted p-values resulting from the contrast comparing the average Ct values from the left ventricle at the time of LVAD implantation (IMP) with the average Ct values from the left ventricle at the time of heart transplant (ELV). There are many miRNAs with significantly different expression values. The GEE model was also used to compare the expression values between the ELV and ERV but there are not as many significantly expressed miRNAs. The results are displayed in Figure 5.

Figure 4.

Histogram of P-values of Comparison between IMP and ELV Based on GEE.

Figure 5.

Histogram of P-values of Comparison between IMP and ERV Based on GEE.

Figure 6 contains the histogram of FDR adjusted p-values resulting from a comparison between the IMP and ELV based on the paired t-test. We can see there are fewer significantly different miRNAs than in the same comparison based on the GEE model. Although the t-test is paired, it does not have the ability to incorporate the additional correlation. The distribution of p-values associated with the comparison between the ELV and ERV time periods based on the paired t-test is similar to Figure 6. The figure containing the distribution is not included in the manuscript.

Figure 6.

Histogram of P-values of Comparison between IMP and ELV based on t-test.

The results from the analysis utilizing the AAA methodology, as well as the Mann-Whitney U test results, are not included here since the distribution of adjusted p-values is similar to the previously discussed t-test.

Discussion

In reviewing the current methodologies, there are statistical models that incorporate the intra-subject correlation created by repeated measurements on the same individuals. The GEE model is a popular method that incorporates the additional correlation. In the exploratory analysis, it is apparent the GEE model also results in additional significantly expressed miRNAs than the paired t-test. Although the statistical model incorporates the additional correlation, what affect does the normalization technique have on the naturally occurring correlation structure? Should the effect be of concern to the analyst? The delta-Ct and the mean normalization techniques shift the mean of the expression value of each plate thus preserving the original correlation structures. The quantile normalization is not a simple shift of the center but actually changes the distribution of the Ct values, resulting in a different correlation structure than naturally occurring. The methodology also reduces the variance based on the coefficient of variation. Based on the readily available software and the cumulative distribution of the coefficient of variation, it appears the quantile normalization technique is the best choice. What affect does the normalization technique have on the GEE model and the resulting significantly expressed genes? How does the effect compare to the shift of center normalization procedures that do not reduce the variation in the Ct values? Should the analysis be performed on the raw, unnormalized data?

The topics discussed require a method to simulate the correlated Ct values. Once the initial problem is solved, then one must evaluate the various combinations of normalization procedures with hypothesis testing procedure to determine the impact on the results.

Conclusions

The motivating example brings to the surface many important questions facing analyst of miRNA Ct values. The increased accuracy and reproducibility of the qPCR methods imply that more researchers are turning to the technology. The experimental designs are becoming more advanced including repeated biologic sampling from the same individuals over time. In general, it is important that the analytic techniques consider the complexities of the experimental design in order to fully understand the results.

The analysis presented is not unique but rather the presentation of an analysis from data preparation, through normalization, and hypothesis testing is. The analysis emphasizes the importance of considering the experimental design. The theoretical formulation of many popular methods is also now contained in one place. In general, it is important that the analysis techniques consider the complexities of the experimental design in order to fully understand the results.

The analysis also brings up several important research questions. The questions represent some of the ideas currently under investigation in order to determine the most appropriate analysis methods for miRNA Ct values obtained from experiments collecting multiple samples on the same individuals over time. The sharing of such information will aid future researchers with the analysis of qPCR Ct values using freely available software and methods.

Appendix

Typically, the delta-Ct normalization technique is derived through the ration of the target gene efficiency (ET) raised to the power

$$\Delta C{t}_T=C{t}_T-C{t}_e$$ (A.1)

and the reference gene efficiency (ER) raised to the power

$$\Delta C{t}_R=C{t}_R-C{t}_e$$ (A.2)

where Ct_T is the Ct value for the target group and Ct_R is the Ct values for the reference group (or control group). The ratio is

$$\frac{E_T^{\Delta C{t}_T}}{E_R^{\Delta C{t}_R}}.$$ (A.3)

If the PCR amplification efficiency achieves the maximum value then both E_T = E_R = 2 and the ratio is written as

$$2^{-\Delta \Delta \mathrm{Ct}}$$ (A.4)

where -ΔΔCt = ΔCt_T - ΔCt_R. The first step, or

$$\Delta Ct=Ct-C{t}_e$$ (A.5)

Is a normalization technique using the endogenous controls as the reference.