The validity of circulating microRNAs in oncology: Five years of challenges and contradictions

Abstract

MicroRNAs (miRNAs) in circulation have received an increasing amount of interest as potential minimal invasive diagnostic tools in oncology. Several diagnostic, prognostic and predictive signatures have been proposed for a variety of cancers at different stages of disease, but these have not been subjected to a critical review regarding their validity: reproducible identification in comparable studies and/or with different platforms of miRNA detection. In this review, we will critically address the results of circulating miRNA research in oncology that have been published between January 2008 and June 2013 (5.5 years), and discuss pre‐analytical challenges, technological pitfalls and limitations that may contribute to the non‐reproducibility of circulating miRNA research.

Highlights

• We performed a comprehensive analysis of published circulating miRNA profiles.

• We analyzed experimental variables of circulating miRNA in different cancers.

• We observed very little overlap between circulating miRNA per tumor type.

• We identified important variables from sample procurement to data analysis.

• These variables contribute to the poor reproducibility of circulating miRNA expression profiles.

1. Introduction

Since the first circulating microRNA (miRNA) signatures were reported as potential diagnostic tools in oncology in 2008 (Lawrie et al., 2008), the scientific literature has seen an exponential growth for these signatures, which can be described as diagnostic, prognostic, or predictive, and have sometimes been reported as a combination of these three characteristics. If these signatures are to have any validity as clinical tools, they should be confirmed by independent groups using identical or similar discovery approaches in the context of a given cancer subtype. In order to evaluate the degree of concordance of these signatures, a comprehensive search of the scientific literature was undertaken. The PubMed search engine was queried by year of publication (from January 2008 to June 10 2013) for the terms “miRNA” or “microRNA” in combination with “blood”, “plasma” or “serum” in the title or abstract. Results were then manually screened to subtract non‐cancer, non‐human signatures and signatures found in normal blood cells as opposed to cell‐free, circulating miRNAs. A total of 154 miRNA expression profiles were identified in 26 different tumor types. We extracted information from individual papers, including the miRNAs claimed to have diagnostic, prognostic, and/or predictive properties; the techniques used to discover and validate these miRNAs; the methods of normalization used; the sizes of discovery and validation cohorts; and other relevant items.

2. miRNA signatures: lack of concordance

The most frequently reported circulating miRNA signatures in oncology are those with diagnostic properties (130 out of 154, or 84%). With these, research groups claim that healthy individuals can be accurately disentangled from individuals suffering from a particular type of cancer. These signatures thus have the potential to be developed into minimally invasive diagnostic tests for the early detection of a wide variety of malignancies. We evaluated the degree of overlap between diagnostic signatures in commonly studied cancers and which are summarized in Table 1, and visualized in Supplementary Figure S1. An overview of all circulating miRNAs signatures is presented in Supplementary Table 1.

Table 1.

Number of miRNA signature per tumor type and overlap between the studies.

Cancer type	Number of signatures	Recurring miRNAs in the signature	Observed in# signatures
Head and neck squamous cell carcinoma	8	miR‐21	5
Gastric cancer	8	miR‐21	3
Pancreatic cancer	6	miR‐21	2
		miR‐196a	2
		miR‐210	2
Prostate cancer	8	miR‐141	3
		miR‐30c	2
		miR‐107	2
		miR‐375	2
		miR‐574	2
Colorectal cancer	11	miR‐29a	4
		miR‐18a	3
		miR‐21	3
		miR‐92a	3
		miR‐7	2
		miR‐17	2
		miR‐20a	2
		miR‐143	2
		miR‐145	2
		miR‐146a	2
		miR‐409	2
Breast cancer	15	miR‐155	4
		miR‐21	3
		miR‐181a	3
		miR‐145	2
		miR‐222	2
Non‐small‐cell lung cancer	8	miR‐21	3
		miR‐145	3
		let‐7f	2
		miR‐20a	2
		miR‐24	2
		miR‐25	2
		miR‐125b	2
		miR‐126	2
		miR‐152	2
		miR‐155	2
		miR‐199a	2
		miR‐223	2
		miR‐320	2

Without a doubt, the cancer type with the most consistency in diagnostic circulating miRNA signatures is head‐and‐neck squamous cell carcinoma, for which four signatures all point exclusively to miR‐21, with a fifth citing the requirement of adding miR‐375 for specificity (Hsu et al., 2012; Komatsu et al., 2011; Kurashige et al., 2012; Tanaka et al., 2012; Wang and Zhang, 2012). Of note, diagnostic specificity of circulating levels of miR‐21 release in circulation correlates not only with head‐and‐neck squamous cell carcinoma and different other tumor types, but also with non‐cancer diseases associated with a strong inflammatory response. MiR‐21 is highly expressed in activated T‐cells and expressed at elevated levels in inflammatory cells in psoriasis (Meisgen et al., 2012), inflammatory bowel disease (Ludwig et al., 2013), as well as activated epithelial cells during wound healing (T. Wang et al., 2012). Therefore, mir‐21 in circulation may reflect strong inflammatory and wound healing‐like responses, rather than a tumor‐specific release of this miRNA.

Fewer than half of gastric cancer signatures center on miR‐21, with five more diagnostic signatures showing no overlap whatsoever (Gorur et al., 2012; Konishi et al., 2012; C. Li et al., 2013; H. Liu et al., 2012; Song et al., 2012; Tsujiura et al., 2010; Wang and Zhang, 2012; Zhao et al., 2010). For pancreatic cancer, tenuous overlap is seen between a four‐miRNA signature (K. Wang et al., 2012; Q. Wang et al., 2012; H. Wang et al., 2012; T. Wang et al., 2012) and three other signatures, which each share one miRNA in common with it but none with each other (Ho et al., 2010, 2012, 2012, 2009). Two other signatures claim diagnostic specificity for the same cancer but act as satellites, sharing none of the previously reported miRNAs (A. Li et al., 2013; Li et al., 2010).

The lack of corroboration becomes more apparent as one analyzes more frequently studied cancers, namely prostate, colorectal, and breast. For prostate cancer, the most commonly reported diagnostic miRNA is miR‐141, but it appears in only three of the eight diagnostic signatures reported so far (Bryant et al., 2012; Z.‐H. Chen et al., 2012; Lodes et al., 2009; Mahn et al., 2011; Mitchell et al., 2008; Moltzahn et al., 2011; Nguyen et al., 2013; Selth et al., 2012). For breast cancer, subtypes are never reported and a degree of variability is thus to be expected. While miR‐155, ‐21, and ‐181a are shared between three signatures or more, the degree of overlap between signatures is poor (Asaga et al., 2011; Cookson et al., 2012; Godfrey et al., 2013; Guo and Zhang, 2012; Heneghan et al., 2010; Jung et al., 2012; Lu et al., 2012; Mar‐Aguilar et al., 2013; Ng et al., 2013; Y. Sun et al., 2012; Wang and Zhang, 2012; Wang et al., 2010; Wu et al., 2012; Zeng et al., 2013; Zhao et al., 2010). Colorectal cancer displays more overlap, but it is generally restricted to miRNAs being shared between only two signatures (Ahmed et al., 2012, 2013, 2012, 2013, 2010, 2013, 2009, 2012, 2012, 2012, 2012, 2012, 2013, 2013). A similarly weak degree of correlation can be observed for diagnostic signatures in non‐small‐cell lung cancer (X. Chen et al., 2012; Cui et al., 2013; Foss et al., 2011; Roth et al., 2012; Sanfiorenzo et al., 2013; Shen et al., 2011; Silva et al., 2011; Wang et al., 2011; Wei et al., 2011; Yuxia et al., 2012).

As with most of the well‐studied tumor types, no single diagnostic signature emerges between these various groups, all claiming specificity and proper validation. This echoes the disconcordance between prognostic messenger RNA (mRNA) expression profiles that have been published over the past two decades. However, in contrast to miRNA signatures, mRNA signatures are normally established using statistical rigor. Methods such as bootstrapping and leave‐one‐out cross validation are in common use in order to control for model over‐fitting; furthermore, most model‐fitting methods reduce or exclude highly correlated expression measures. Hence, very different signatures can often have a very similar biomarker value. In all miRNA studies that we examined, all differentially expressed miRNA are considered to be part of the signature without exclusion of any miRNA. These are lists, rather than signatures.

3. miRNAs in circulation and tissue

One of the many underlying assumptions in the hunt for circulating miRNA signatures is that the release of these miRNAs in the blood stream is altered in a specific manner by a particular disease state. The most obvious hypothesis is that tumor‐specific miRNAs escape from malignant cells and end up as detectable ejecta in circulation, a similar principle being actively pursued in the detection of circulating tumor cells. If this hypothesis is correct, one could correlate a particular miRNA's overabundance or rarity in the blood of a cancer patient with a similar overabundance or rarity in that patient's tumor tissue. Such a correlation would take us closer to validating the significance of these miRNA changes, as they would be shown to be direct consequences of tumor formation and progression.

It is thus unfortunate to discover that only 49% (75/154) of published circulating miRNA signatures in cancer research are accompanied by a study of the expression of the miRNA signatures in the corresponding tissue, either through direct experimentation or through literature search. Of the 154 signatures studied, only 7% displayed perfect correlation in a discovery setting, meaning that for every significant miRNA reported in circulation after an unbiased discovery (i.e. miRNA levels were tested with complete disregard for their previously reported levels in the corresponding cancer tissue), its levels were seen as moving in the same statistically significant direction in tissue. This percentage is identical to that of signatures in which a complete contradiction was seen, meaning that, while there was also statistical significance between the tissues being compared (healthy vs. malignant, stage I vs. stage IV), each miRNA was going in the opposite direction in the tissue, such that a miRNA overexpressed in the circulation of breast cancer patients was actually underexpressed in cancerous breast versus normal breast tissue! In 3% of signatures, there was no significant difference between the levels of these reported miRNAs in the compared tissues. In 22% of signatures, there was correlation, but these circulating miRNAs were not part of a discovery protocol, but were rather studied in circulation after their dysregulation in tissue had been reported in the literature.

The often observed lack of correlation between miRNA expression in circulation and tissue is puzzling when assuming that circulating tumor‐specific miRNAs are actually secreted by tumors, and that the ability to quantify miRNAs is not determined by efficacy of miRNA extraction from tissue, serum or plasma. Given the lack of concordance between most miRNA signatures in a given tumor type and the mixed correlations between circulating and tissue‐specific levels of these reported miRNAs, there is a dire need for a comprehensive exploration of the challenges of circulating miRNA experiments. These difficulties, which range from pre‐analytical challenges such as the proper collection and isolation of blood derivatives and the miRNAs they contain, to the post‐analytical challenges of data standardization and normalization.

4. Pre‐analytical challenges of circulating miRNA experiments

Pre‐analytical variables can be defined as the elements of a test which can affect the composition of the sample to be tested. They can generally be divided into patient factors, sample collection, and sample handling. In the case of molecular biology, we can add to this list the process of nucleic acid extraction itself, since the assay is generally conducted on nucleic acids and not on the body fluid. The question of how miRNAs end up in circulation is an important one, since its answer may shed light on the patient factors that need to be controlled to ensure the reproducibility of such an assay. We will look at the quantity of RNA in the blood, the way in which circulating miRNAs have been hypothesized to be protected from degradation while in the blood, the differences between serum and plasma, possible methods of small RNA enrichment in extraction protocols, and whether miRNAs isolated in circulation are truly cell free.

4.1. Source

The principal advantage of isolating miRNAs from biological fluids (e.g. blood, saliva, urine) is the ease of sample procurement: a useful miRNA signature in blood has the potential of becoming a minimally invasive diagnostic tool that can be relied on repeatedly for monitoring purposes during the course of the disease, thus avoiding more invasive procedures such as biopsies. Both serum and plasma have been used in the quest for circulating miRNA signatures in cancer, with neither being preferred. Indeed, 42% of signatures have been reported in plasma and 57% in serum, with an additional 3 signatures out of 154 using both. These trends are consistent over the years, with neither blood derivative being adopted as the optimal medium for such assays.

Unlike biopsied or surgically resected tissue, though, biological fluids are thought to contain much lower levels of miRNAs, although exact numbers are hard to come by. El‐Hefnawy et al. (2004) calculated a plasma RNA concentration in the range of 1–10 ug/L using quantitative polymerase chain reaction (qPCR), while Weber et al. (2010) detected a median value of 308 ug/L in the same medium. In contrast, K. Wang et al. (2012) showed that total RNA concentration in serum (62 ± 26 ug/L, 12 patients) was superior to RNA concentration in plasma (36 ± 17 ug/L). However, the accuracy to quantify miRNAs in small volumes is subject of debate, and that the range of concentrations of miRNAs in healthy individuals is often large (Kroh et al., 2010; Mitchell et al., 2008). It is important to realize that miRNA in platelets can be released upon coagulation, and that miRNAs released from hemolytic erythrocytes (Willeit et al., 2013; Kirschner et al., 2011), such as miR‐16 that is often used as a normalizer (see below) may explain these different values for total miRNA concentration and may increase variability in miRNA analyses. Any controllable and non‐controllable variation in the preparation of blood samples will therefore impact the outcome of miRNA expression results (Cheng et al., 2013). Low levels of a particular analyte can cause difficulties in detection while minute changes in expression which would have been deemed insignificant in a higher concentration range can appear quite dramatic (Armbruster and Pry, 2008), especially when the mode of detection is PCR amplification which works exponentially.

4.2. Circulating or contaminating miRNAs?

The method of obtaining the serum or plasma sample is a potential source of variability (Kroh et al., 2010; Cheng et al., 2013). Numerous variables surrounding phlebotomies influence the miRNA quantification result such as the type of anticoagulant coating, the type of blood tube used, diurnal variations in miRNA expression and release, the state of fasting of the individual, blood cell count, and needle gauge (Kroh et al., 2010), as well as hemolysis (Kirschner et al., 2011). Moreover, white blood cells from the buffy coat and cells present in the skin plug obtained at venipuncture could in theory contaminate a sample: miRNA concentrations being several fold higher in cells than in circulation, the signal from the cellular miRNAs could easily “drown out” the signal coming from the circulatory miRNAs. Indeed, the plasma levels of circulating miRNAs reported as solid tumor biomarkers actually correlate well with blood cell counts (Pritchard et al., 2012). The variation in miRNA biomarker levels due to blood cell effects may be greater in magnitude than differences reported between cancer patients and controls. Pritchard et al. (2012) rightfully question the nature of what it is that we are detecting when we claim to identify freely circulating miRNAs, and more work is needed to clarify whether or not these so‐called circulating miRNA signatures are actually measuring levels of blood cell miRNAs.

4.3. Extraction methods

The manner in which circulating miRNAs are extracted from biological fluids can also have an impact on their ultimate detection. Life science companies are now marketing RNA extraction kits which allow one to isolate miRNA‐enriched fractions. The researcher interested in circulating miRNAs is thus faced with a choice: total RNA or miRNA‐enriched fraction? Total RNA is clearly favored: 71% of reported signatures were clearly based on total RNA extraction from either serum or plasma, while only 5% of signatures made use of a small‐RNA‐enriched fraction. Three signatures were based on unprocessed blood derivatives, while a non‐negligible 22% of signatures were not accompanied by a clear enough RNA extraction protocol that would identify the type of RNA used. The discarding of a large‐RNA fraction may carry with it the risk of losing miRNAs attached to longer RNA molecules and of thus skewing the distribution of recovered miRNAs. This is suggested by a study from 2010 in leukocytes reports that small RNA yield quantification did not show reproducible results in three different methods, with the authors hypothesizing that the filters were not optimized for small‐RNA binding (Hammerle‐Fickinger et al., 2010). The use of total RNA extraction, however, is not a no‐brainer remedy to the situation; it has been shown that, while the concentration of total RNA recovered using three different methods may be similar, the corresponding concentration of lower‐molecular‐weight‐fraction RNA is substantially different (Masotti et al., 2009). The authors go on to show that TRIzol extraction yielded the highest proportion of small RNA species to total RNA (22–34%), with the use of QIAGEN's RNEasy Mini Kit resulting in the lowest (2.5–3%). It thus appears critical to keep using a single extraction method within an experiment; failing to do so could potentially be remedied by stringent normalization, which will be discussed under post‐analytical challenges.

Regardless of the type of nucleic acid isolation, centrifugation has a strong impact on miRNA recovery. It was recently demonstrated that a two‐step centrifugation process during blood phase separation reduces the number of miRNAs collected from plasma compared to a one‐step centrifugation, and the authors hypothesized that miRNA‐containing particles may clump together and precipitate at high centrifugal speeds (Zheng et al., 2013). Furthermore, the effect of centrifugation as a preclearance step is sample type and miRNA dependent. Whereas a 15,000 g centrifugation step of plasma reduced the concentration of miR‐15b, miR‐24, and miR‐16 as measured by qPCR (mean increase in Cq 7.9, 7.9 and 2, respectively), this loss of miRNAs was much less for centrifugation of serum samples and, importantly, consistent for the three miRNAs with only a 1 Cq value increase. (McDonald et al., 2011). A reduction of miRNA yield as a result of centrifugation of plasma has been reported in other studies (e.g. Page et al., 2013; Zheng et al., 2013). Important to note is that centrifugation of serum or plasma did not affect the recovery efficiency of spiked‐in synthetic Caenorhabditis elegans miRNA (cel‐miR‐54) (McDonald et al., 2011). Therefore, using a spiked‐in positive control to determine isolation efficiency for plasma does not reflect that of the endogenous miRNAs in this sample type. This difference is most likely explained by the fact the spike‐in synthetic miRNA is not contained within exosomes and protein complexes that are pelleted in the centrifugation step.

The difference in miRNA stability in serum and plasma needs to be considered. In serum, the concentration of miRNAs remain relatively stable with a minor, yet significant increase of Cq values (0.5–1.1) (McDonald et al., 2011) and is in line with earlier observations (Mitchell et al., 2008). However, increasing the time between sample procurement and processing and storage and processing at room temperature instead of 4C negatively affects miRNA yield form plasma samples (Page et al., 2013).

4.4. MiRNA pre‐amplification

A preamplification step is recommended by Applied BioSystems (TaqMan) for very small miRNA quantities. However, TaqMan preamplification affected serum miR‐200a and miR‐200b levels, but not those of miR‐1290 in a screen to identify diagnostic markers for pancreatic cancer (A. Li et al., 2013). Similarly, we also observed miRNA specific non‐linear amplification: e.g. whereas miR‐1298 was strongly amplified, only a mild increase in relative expression for miR‐518b, and little to no impact on the relative expression of miR‐1233 was observed (our unpublished results). In contrast, Van Schooneveld et al. (2012) did not encounter this problem, showing an acceptable correlation between 439 preamplified and non‐preamplified miRNAs. These inconsistencies demonstrate yet unknown or ill‐defined variables that contribute to preamplification reproducibility.

A confounding factor in miRNA expression analysis may be the reverse transcription reaction (RT) itself, which prefaces the preamplification. Tjensvoll et al. (2012) have previously reported differences in the detection of miRNAs depending on whether or not they used singleplex or multiplex RT. MiRNAs which were not detected on the TaqMan array, for which multiplex RT was used, could be detected using the same technology when a singleplex RT was conducted for validation purposes. The authors hypothesize that the efficiency of cDNA synthesis may be impacted by the number of targets being simultaneously transcribed and that rarer miRNAs may be outcompeted in a multiplex format. The large number of publications describing profiling of circulating miRNAs often does not address the RT reactions in detail, leaving many possible confounding variables unknown.

Understanding and controlling pre‐analytical variables is thus crucial to help ensure the reproducibility of circulating miRNA experiments but, even with such forethought, the question of what it is that is being isolated from serum and plasma (i.e. released from tumor, white blood cell, platelet or erythrocyte‐derived) remains open.

5. Analytical challenges of circulating miRNA experiments

The main analytical variable at work is the choice of a detection platform. A systematic comparison of three most frequently used platforms for the detection of circulating miRNAs (GeneChip miRNA 2.0 Array (Affymetrix), miRCURY Ready‐to‐Use PCR Human Panel I+II V1.M (Exiqon), and TaqMan Human MicroRNA Array v3.0 (Life Technologies) demonstrated differences in the limits of detection (Jensen et al., 2011). The GeneChip array was shown to not be reliable enough for the detection of plasma RNA, while Exiqon's miRCURY locked‐nucleic‐acid (LNA) SYBR green technology platform was more sensitive than Life Technologies' TaqMan hydrolysis probe platform for the detection of low miRNA levels (50‐to‐200 copies and below). Furthermore, TaqMan and Exiqon technologies displayed poor consistency in a direct comparison of miRNA expression in serum and plasma sample from 4 healthy individuals (K. Wang et al., 2012). Out of the 358 miRNAs tested by both platforms, only 67 were detected in all four samples with both platforms, with the Cq values obtained by TaqMan being higher by 6.7 units on average compared to miRCURY. While these studies show to that LNA probes are more sensitive than TaqMan hydrolysis probes, the price tag LNA probes command can be intimidating, especially in an array format. This is probably the reason why only 6% of signatures made use of LNA probes, all of which were used in the discovery phase and never to validate findings. While an array using LNA technology may prove cost prohibitive to an academic researcher, such probes would prove useful in the validation phase given what we now know of their sensitivity.

By far the most commonly used technology for detecting miRNAs in circulation is qPCR using TaqMan hydrolysis probes (Figure 1). This technology was used 49% of the time in the discovery phase and 80% of the time when a validation was performed. By comparison, the more sensitive LNA probes were used in 9 discovery studies (6%) and in 1 validation study (1%). qPCR making use of SYBR Green with non‐LNA‐primers was fairly popular, used in 14% of the discovery and in 14% of validation assays, while microarray‐based methodologies represent 16% of the discovery assays.

Figure 1.

Frequency of platform choice for circulating miRNA in discovery and validation. Pie charts showing the use of different miRNA detection platforms for the discovery and validation of circulating miRNAs.

More recently, a DNA‐RNA hybrid capture technology has been marketed by nCounter NanoString technologies, which is capable to detect miRNAs as well. Whereas sample preparation does not involve an amplification step as such, the miRNAs need to ligated to miRtag for quantification, which involves a PCR step whose efficacy is determined by miRNA quality and purity. This technology was used in one study (Suryawanshi et al., 2013) to demonstrate lack of concordance between miRNA expression in circulation and tissue, but a comparison between miRNA profiles obtained with NanoString and qPCR was not made.

Next generation sequencing technology has opened up new possibilities to detect miRNAs. Whereas hybrid capture and qPCR technologies only work for those pre‐ and mature miRNAs that are known and for which a primer pair and probe is designed, unbiased deep sequencing will identify all known and unknown miRNAs (miRNA‐seq), including single nucleotide variations that may affect their activity. However, miRNA‐seq data analysis requires a thorough investigation of the raw data to determine and quantify the actual expression of mature miRNAs and accurate prediction of the activity of novel miRNA based on the secondary structure of the pre‐miRNA and the miRNA target prediction. Normalization of miRNA expression remains a challenge as true for any platform (Garmire and Subramaniam, 2012; Zhou et al., 2013; and section below) For the identification of novel miRNAs, an integrated application tool has been developed and is freely available (An et al., 2013).

5.1. Post‐analytical challenges of circulating miRNA experiments

The importance of selecting the right detection method cannot be underestimated, but the next question one needs to ask is what to do with the raw data. Data standardization and normalization, as well as calculations of statistical significance, are often hastily examined and conducted in suboptimal ways.

It is of particular interest to note that a large fraction of the scientific literature does not release raw Cq values when reporting circulating miRNA signatures, preferring instead to show relative expression values. This makes it difficult for reviewers in particular and the scientific readership in general to assess the potential reproducibility of these signatures: indeed, while there seems to be no consensus, Cq values between 30 and 35 are usually used as threshold values of reliability, so that any target Cq value above this range should be interpreted with great caution. Similar to quantitative analysis of mRNA expression, Minimal Information for publication of Quantitative real time PCR Experiments (MIQE guidelines; Bustin et al., 2010, 2011) should be provided for PCR‐based miRNA expression. For hydrolysis probes (TaqMan, Applied Biosystems), this is of concern because the company is reluctant to release the proprietary sequences of the primers/probe combination.

With qPCR, the most important issue remains that of data normalization, i.e. the adjustment of raw data to render it comparable between samples. Since extraction, reverse transcription, and amplification efficiencies can vary between the samples of a given experiment, a normalizer must be selected against which to judge disparities between samples. In traditional mRNA qPCRs, a common normalizer is a so‐called “housekeeping gene”, such as GAPDH, HPRT, PPIA, RPLP0 or combinations thereof, the expression of which is assumed to be constant in different individuals. In miRNA qPCR, however, disagreement is seen over the putative existence of “housekeeping miRNAs”. Inarguably, miR‐16 has emerged as the frontrunner for the position, being used as the normalizer in 24% of discovery phases of circulating miRNA signatures reported in oncology (Figure 2). Most research groups used miR‐16 as the de facto normalizer of choice, ignoring the growing body of research that shows that it can be dysregulated in diseases including cancer (Bhattacharya et al., 2009; Bonci et al., 2008; Cai et al., 2012; Chen et al., 2013; Gao et al., 2012; Musumeci et al., 2011; Navarro et al., 2010; Rivas et al., 2012; Sun et al., 2013; H. Wang et al., 2012; Zuo et al., 2011).

Figure 2.

Frequency of miRNA expression normalization methods in discovery and validation. Pie charts showing the use of different normalization strategies for circulating for circulating miRNA in discovery and validation.

An alternative to normalizing to miR‐16 is to identify a set of miRNAs that display stable expression with minimal variability across the whole data set, using an algorithm such as genorm^PLUS (Biogazelle). If the number of samples tested on the array is sufficient, stable miRNAs can be identified and tested alongside miRNAs of interest in validation experiments. Such normalization using a combination of stable miRNAs was conducted in 7 out of 156 discovery phases (4%) and in 5 out of the 67 validation experiments (7%) in which the choice of normalization method was clearly stated. This approach however can only be held accountable for similar samples—a normalizing pair found in colorectal cancer could not be used with melanoma samples without validating it in an array experiment first—and this approach is difficult to adapt for groups wanting to bypass arrays and directly investigate specific targets.

A common normalization method is to use small nucleolar RNAs (snoRNAs) in the belief that their levels are independent of health status. Indeed, 29/151 (19%) of discoveries and 20/67 (30%) of validations for which normalization was defined had their raw data normalized against a single or small group of snoRNAs, such as RNU6B and RNU48. Like miR‐16, however, these small nucleolar RNAs, once thought to be immune to morbidities, have been implicated in disease and should therefore not be used as de facto normalizers unless shown to be stable in the sample cohorts being studied (Dong et al., 2008; Liao et al., 2010; Mourtada‐Maarabouni et al., 2009).

The ease with which normalization can be confused with standardization, however, can be shown by the number of signatures (12% in discovery and 18% in validation) in which normalization was effected using a spike‐in miRNA. This methodology, which can be used to correct for differences in RNA recovery and qPCR efficiency, works by adding a standard amount of a non‐human synthetic miRNA (e.g. C. elegans and Mus musculus being popular choices) in the plasma or serum solution at the beginning of RNA extraction. What it should not be used for is to account for differences in endogenous miRNA expression and release between samples. If a spike‐in miRNA is used for standardization purposes, an independent means of proper normalization is still needed.

A further note about normalization applies to large‐scale evaluations of expression. Statistical normalization methods commonly used in genome‐wide mRNA studies assume that a large proportion of the genes are not expressed, but this assumption is unlikely to be true for miRNAs. Hence, well‐accepted methods such as quantile normalization may be inappropriate for normalizing miRNA levels (Qin et al., 2013).

6. Power and replicability

There are several statistical issues that should be considered when evaluating whether miRNA expression differences are real and important. The first issue is power. One very common reason for a lack of replicability between experiments is simply due to insufficient power, i.e. sample sizes that are too small (Gorroochurnet al. 2007). Significant results from an initial study can be described by an estimate of the magnitude of the difference between groups, together with the confidence interval for the plausible range of values for the difference. Follow‐up studies should ideally be designed to have enough power to detect a difference corresponding to the smaller end of the confidence interval, not to the centre. In all studies that we have analyzed for this review, none provided any type of power analysis to justify the number in the validation assay.

Secondly, whenever a large series of measures is tested, the results identified as “most significant” in the study are subject to winner's curse (Ioannidis, 2008; Faye et al., 2011). This means that the most significant results will be biased away from the null hypothesis, simply by selecting and reporting the top findings. It follows that in any follow‐up study, the measures that were most significant in the first study, even if truly associated with phenotype, are expected to demonstrate a smaller differences in the new data. Hence, in general, sample sizes for replication need be much larger than sample sizes for discovery (Gorroochurnet al. 2007). There are methods to estimate and reduce winner's curse bias (Sun et al., 2011); these should then be used to calculate sample sizes for replication.

A third issue worth considering is the choice of significance thresholds. Many different miRNA measures may display some correlations or operate in the same network. In such a situation, members of such a group may show small differences in expression that do not meet a Bonferroni‐adjusted significance threshold, despite evidence of a consistent trend. In such situations, considering the false discovery rate (FDR) instead of the p‐value can be useful (Benjamini et al., 2001). It should be noted, however, that the FDR enables identification of a larger number of truly associated measures by relaxing the significance threshold, thereby accepting a higher proportion of false associations among the results selected for further evaluation.

If obtaining a large enough validation study is difficult, another approach to obtaining more accurate estimates of the true difference in expression is to perform meta‐analyses of the magnitude of the expression differences across several independent studies (Ioannidis et al., 2001; Skol et al., 2006). The meta‐analysis, ideally, should be performed genome‐wide. Alternatively, the “significant” hits from the initial study will need to undergo bias correction to protect against the winner's curse mentioned above.

7. Conclusion

Between January 2008 and June 2013, a total of 154 diagnostic, prognostic, and/or predictive circulating miRNA signatures in the field of cancer research alone have been described. However, the overlap between tumor‐type specific signatures (i.e. top differentially expressed miRNAs) is poor and this may be the results of yet undefined confounding factors, and uncontrolled or difficult to control known variables. The lack of adherence to MIQE guidelines of almost all studies makes a retrospective analysis of the results, as well as true duplication thereof in an independent setting difficult, if not impossible.

This questioning of the validity of the signatures is part of a larger interrogation slowly being leaked in the literature. After claiming surprise at the lack of overlap between their newfound signature in breast cancer and previously reported miRNAs, the team of Cookson et al. (2012) reported that circulating levels of abundant tumor miRNAs were unrelated to the presence of tumors. The group goes on to propose that passive shedding of miRNAs by tumors is probably wrong, and that their release in circulation must be through an active and carefully regulated process. A report from Leidner et al. (2013) demonstrated little association between their breast cancer data set of 1145 miRNAs and that of another group using identical methodology, demonstrating uncontrolled variables in the search for miRNAs as biomarkers.

It is easy to be reminded of the gold rush toward diagnostic mRNA signatures ten years ago. A new, seemingly comprehensive technology was adopted by a number of researchers in search of a pattern of gene expression dysregulation to predict disease state. Today, only one of these signatures has been commercialized and is being used by a subset of oncologists: Oncotype DX, which has prognostic and predictive significance for women with early‐stage, estrogen‐receptor‐positive breast cancer. The rush toward reporting correlative signatures of cell‐free miRNAs in the blood may be premature in light of the gaps in our knowledge of their biology and technical challenges of their identification.

Supporting information

The following are the supplementary data related to this article:

Figure S1 Venn diagrams visualizing shared miRNAs in different signatures. For each tumor type (A: head and neck squamous cell carcinoma, 8 signatures; B: pancreatic cancer, 6 signatures; C: gastric cancer, 8 signatures; D: prostate cancer, 8 signatures; E: colorectal cancer, 11 signatures; F: breast cancer, 16 signatures; G: non‐small cell lung cancer, 8 signatures) a signature is represented by a colored form. The miRNA(s) shared between signatures is identified. If the signature contains more miRNAs than the one identified, then a number for the additional miRNAs in that signature is given. Guide to interpretation. A) Off all 8 circulating miRNA signatures in head and neck squamous cell carcinoma, 3 signatures did not show any overlap. These signatures were characterized by 1,1 and 7 miRNAs. Six signatures all showed miR‐21 expression, with one of these having one more miRNA. B) Off all 6 signatures, 2 signatures (one with 1 miRNA and the other one with 2 miRNAs) did not show any overlap with the other 4 signatures. miR‐21, miR‐196a and miR‐210 plus one more were present in one signature, but miR‐21, miR196a and miR210 were detected in only 2 signatures. The diagrams in C‐G should be read according to these same guidelines.

Supplementary data

Supplementary data

Supplementary data 1.

Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.molonc.2014.02.009

Jarry J., Schadendorf D., Greenwood C., Spatz A., van Kempen L.C., (2014), The validity of circulating microRNAs in oncology: Five years of challenges and contradictions, Molecular Oncology, 8, doi: 10.1016/j.molonc.2014.02.009.