Vigilance and Validation: Keys to Success in RNAi Screening

Abstract

In the twelve years since the process of RNA interference (RNAi) was first discovered, great progress has been made in understanding its mechanism and exploiting its ability to silence gene expression to study gene function at a genome-wide level. Its extensive use as a screening method has yielded many published lists of genes that play novel roles in higher eukaryotes. However, the usefulness of this information is potentially limited by the occurrence of unintended off-target effects. Here we review the potential causes of off-target effects, and the impact of this phenomenon in interpreting the results of high-throughput RNAi screens. In addition to targeting the intended gene product, artificial short interfering RNAs (siRNAs) can produce off-target effects by down-regulating the expression of multiple messenger RNAs through microRNA-like targeting of the 3′ untranslated region. We examine why this phenomenon can produce high hit rates in siRNA screens, and why independent validation of screening results is critical for the approach to yield new biological insights.

Introduction

The effects of RNA interference (RNAi) were first observed in 1990 in plants (petunia) as an unintended consequence of transgene overexpression (1, 2). The underlying mechanism remained obscure for eight years until the pioneering work of Fire, Mello and colleagues demonstrated that double stranded RNA was critical for suppressing gene expression in C. elegans (3). The availability of the sequenced C. elegans genome, as well as the simplicity of performing RNAi experiments in this organism, led to the first systematic, chromosome-wide analysis of gene function using this method (4, 5), and subsequently to genome-wide screens (6-10). Many screens have also been performed using Drosophila cell culture systems (11-14), where it is straightforward to induce RNAi by treating cells with long double stranded RNA molecules.

Potential adoption of these methods to mammalian cells was limited by the fact that long double stranded RNAs potently activate the innate immune system (15-19), an ancient antiviral response. However, once it became clear that short double stranded RNAs (or short interfering RNAs; siRNAs, 21 nucleotides) could induce gene silencing without strongly activating this response (20), siRNA-based methods have become widely adopted as a tool for studying gene function. Development of libraries of chemically synthesized siRNAs or plasmid-encoded short hairpin RNAs (shRNAs) has enabled the execution of many genome-wide screens in mammalian cells (for reviews, see (refs. 21-25). Despite the transformative potential of this new technology, it is important to recognize that suppression of gene expression by RNAi is not equivalent to inactivation of a gene by mutation. Unlike genetic approaches that modify the DNA of an organism, RNAi acts at the level of mRNA, decreasing mRNA levels or the ability of the mRNA to be translated. As a result, RNAi-based experiments can suffer from a lack of sensitivity due to incomplete suppression of gene expression, or a lack of specificity due to suppression of unintended genes. Here we review the potential challenges of RNAi-based screening, the need for vigilance in designing and interpreting a screen, and the importance of independent validation of screening results. We also examine some potential reasons for the relatively low rate of hit validation from published RNAi screens.

A convergent machinery for the siRNA and microRNA pathways sets the stage for off-target effects

Because experimentally induced RNAi-based gene inactivation co-opts the endogenous cellular RNAi machinery, it is important to understand the normal functions carried out by the RNAi pathway to appreciate how off-target effects can arise. RNAi-based gene regulation can be broadly classified as siRNA-based or microRNA-based (Figure 1; see (26, 27) for review). The key physiological distinction between these pathways is that siRNAs typically silence the expression of the genes from which they are derived, whereas microRNAs typically silence the expression of heterologous genes. For example, the siRNA pathway is important in host defense to suppress the expression of genes encoded by an invading virus. In this case, the siRNAs are derived from double-stranded RNA intermediates that arise during viral replication, and the target RNA is the sequence from which the siRNAs are derived. In contrast, microRNAs are important for normal regulation of gene expression in cells, and downregulate the expression of genes located at loci distinct from the site of expression. However, once the small RNA is generated, the pathways rely on a common downstream machinery to inhibit gene expression. This overlap in function is a key reason that exogenous siRNAs used as research tools can induce nonspecific microRNA-like effects, reducing the expression of unintended genes.

Figure 1.

Model of mammalian RNA interference pathways.

RNA molecules involved in RNAi pathways originate from artificial sources including siRNA transfection and shRNA expression from a transfected vector or endogenous sources where miRNA genes encode a pri-miRNA. A protein complex Drosha/Pasha cleaves the pri-miRNA to generate one or more pre-miRNA molecule. shRNAs and pre-miRNAs are transported into the cytoplasm by a mechanism involving Exportin5. The RNase III enzyme Dicer further cleaves shRNAs and pre-miRNAs into mature siRNAs and miRNAs, respectively. siRNA and miRNA single strands incorporate into RISC complexes and serve as templates to target mRNAs for cleavage, translation inhibition and/or mRNA decay. The fate of the targeted mRNA depends on the extent of sequence pairing.

In the siRNA pathway, double stranded RNA that is expressed by an invading virus, or that is expressed by convergent transcription from the genome, is processed by the enzyme Dicer into short 21–22 bp double stranded siRNAs. Next, one strand, called the guide strand, is preferentially incorporated into RISC (RNA-induced silencing complex), and the complementary strand is degraded. However, because strand choice depends on thermodynamic stability of the ends of the double-stranded RNA, in some cases either strand may be incorporated into RISC, leading to a chance of targeting unintended messages. The guide strand then targets RISC to complementary mRNAs, and the Argonaute protein of RISC (Ago2 in mammals) cleaves the target mRNA, rendering it susceptible to degradation by exonucleases. Cleavage by Argonaute requires perfect sequence complementarity at the site of cleavage, 10 bp from the 5′ end of the guide strand. Because an siRNA induces endonucleolytic cleavage of the target mRNA, it can strongly reduce gene expression.

In the microRNA pathway, the genes encoding a microRNA are transcribed by RNA polymerase II and they are capped and polyadenylated like other Pol II transcripts. The resulting transcript, which contains one or more hairpin structures, is processed by the enzyme Drosha in the nucleus, and the resulting product is exported to the cytoplasm, where it is processed by Dicer to yield a mature microRNA. At this point, the siRNA and microRNA pathways converge, as a single strand of RNA derived from the microRNA is incorporated into RISC and guides the selection of silencing targets. However, because microRNAs typically do not have perfect sequence complementarity with their targets, RISC does not generally induce cleavage of the target mRNA. Instead, translational repression is the main outcome, although in many cases the target mRNA is also destabilized by mRNA decay (28). In contrast to the siRNA pathway where complete sequence complementarity is required for target mRNA cleavage, microRNA-induced effects typically require a short region of homology between the microRNA and its target (Figure 2). This region, referred to as the “seed region”, is present at the 5′ end of the guide strand of the microRNA, typically extending from nucleotides 2–8 (29), though recent work indicates that a somewhat longer seed region in the center of the microRNA can lead to mRNA target cleavage in a manner reminiscent to siRNA-directed cleavage (30). As a consequence, targeted genes need only contain a short region of sequence homology to the microRNA, typically seven to eight nucleotides in length. Because of this limited sequence complementarity, a given microRNA has the potential to target a large number of different mRNAs in the cell. The target sites for most endogenous microRNAs are located in the 3′ UTRs of target genes, perhaps because microRNA-RISC complexes are not easily displaced by the translocating ribosome. Repression of gene expression by a microRNA-containing RISC complex is generally not as complete as that induced by a perfectly-matched siRNA-containing complex, perhaps leading to only 50% downregulation (29, 31). Cellular mRNAs may also be targeted by multiple microRNAs, thereby enhancing downregulation of the message. The microRNA pathway is known to regulate many different cellular processes (32, 33). Over 1000 different microRNAs are expressed in human cells (34), and it has been estimated that each microRNA may have an average of 300 different targets, with over half of human protein-coding genes containing conserved microRNA target sequences (35). Because microRNA-based regulation has such a broad reach in cellular biology, its perturbation has the potential to affect almost any type of cellular screen or experiment that might be performed.

Figure 2.

Seed sequence-based interaction of siRNA guide strand with an off-target mRNA.

The 7-nucleotide seed region of the siRNA guide strand (position 2–8) shows full complementarity (blue, A and B) to a target site in the 3′UTR of off-targeted mRNA. Additional siRNA 3′ sequence pairing (green, B) to the target mRNA may contribute to efficient off-targeting by the siRNA. Centered complementarity site (siRNA position 4– 14 or 5–15) to the mRNA target (red, C) can drive a RISC slicer activity-dependent cleavage of the target mRNA.

Sequence-independent off-target effects

There are several mechanisms through which introduction of an exogenous siRNA or shRNA may broadly affect the physiology of a cell, independent of the specific sequence of the reagent used (see (36) for review). First, exogenous expression of short hairpin RNAs (shRNAs) can interfere with endogenous processing of microRNAs. For example, sustained expression of an exogenous shRNA in mice can be fatal, perhaps as a consequence of saturating the pathway that is used to export miRNA precursors from the nucleus (37). Second, introduction of siRNAs into a cell can displace endogenous microRNAs from RISC, thereby altering normal patterns of gene expression. In support of this idea, bioinformatics analysis of published transcriptional profiling experiments revealed that the predicted targets of endogenous microRNAs are expressed at higher levels following transfection of an siRNA designed to target a discrete gene (38). Third, even when using short double stranded RNAs, cells may mount non-specific immune response to transfected synthetic double strand RNA (dsRNA) (39) or viruses used to express shRNAs (40). Fourth, siRNAs have been shown to induce a type of cell-stress response when used at high concentration (41, 42). Therefore, investigators need to be aware that introduction of an exogenous siRNA or shRNA molecule may alter the physiology of a cell in a manner independent of the specific sequence of the siRNA or shRNA.

Sequence-specific off-target effects

In addition to the potential for an exogenously-introduced siRNA or shRNA to globally perturb miRNA-based gene silencing in the cell, each individual siRNA or shRNA may downregulate the expression of a subset of genes in a sequence-dependent manner by acting like a microRNA. In this case, in addition to inducing cleavage of the intended target via perfect base-pairing and Argonaute-induced cleavage, the siRNA may induce microRNA-like effects via interaction with target sequences through its seed region (26, 43-45) (Figure 2). Because this phenomenon requires only a short region of sequence homology, each siRNA has the potential to nonspecifically downregulate the expression of hundreds of different genes in the cell.

Transcriptional profiling has proved to be a useful tool to identify off-target effects at a genome-wide level. However, this approach only detects changes in mRNA level, and therefore may miss off target effects that act only by suppressing translation. In 2003, the first evidence of microRNA-like off-target effect was reported (46). In this study, a series of siRNAs were designed to target two different genes, MAPK14 and IGF1R. Transcriptional profiling revealed that each of the siRNAs produced a distinct pattern of effects on transcription. Though the intended target was downregulated, the expression of many other genes was also affected, in a range of 1.5-fold to 4-fold. This effect was first observed at a high concentration (100 nM) of siRNA, but persisted when the dose was reduced to 4 nM. Kinetic analysis indicated that these changes occurred early and prior to any changes in protein levels of the intended target, suggesting they were not an indirect consequence of knockdown of the intended protein. Analysis of the downregulated genes indicated that as few as eleven contiguous nucleotides of identity to the siRNA were sufficient to induce downregulation, consistent with a microRNA-like effect.

A similar conclusion was reached by a study that analyzed the transcriptional profile of cells treated with 12 different siRNAs targeting three genes (47). This study also used siRNA at 100 nM concentration, and identified 347 genes that were downregulated by only one siRNA but not by other siRNAs targeting the same gene, suggesting they were off-target effects. Similar results were obtained when the concentration of siRNA was reduced to 50 nM. Analysis of the off-targeted genes indicated the frequent presence of one or more perfect sequence matches within the 3′ untranslated region (UTR) matches to the hexamer or heptamer seed region (positions 2– 7 or 2–8) of the guide strand of the siRNA, consistent with a microRNA-like off-target effect. The importance of the seed match region in mediating these effects was confirmed in a later study (48), as alteration of the seed region of the siRNA eliminated the off-target effects but created a new set of off-targeted genes. This study also demonstrated that seed-match mediated off target effects can also arise from expression of shRNAs in cells, and is not restricted to the effect of transfected siRNAs.

Although the evidence for the ability of siRNAs to induce microRNA-like off target effects is strong, it is important to point out that not all studies have reached this conclusion. Whereas one study identified dose-dependent off target effects of an siRNA targeting GFP (49), another transcriptional analysis of cells treated with siRNAs targeting GFP showed no reproducible nonspecific changes in gene expression (50). Other studies have suggested that when a lower concentration of siRNA is used (20 nM), the effects can be quite specific (42). In an analysis of five siRNAs targeting the Rb1 mRNA, transcriptional profiling revealed that the expression of 903/919 genes was altered by three out of five siRNAs; 720 were regulated by all 5 siRNAs (42). Another recent study indicated that selecting for siRNAs with most potent specific knockdown allows using the siRNAs at low nanomolar concentrations which reduced the extent of off-target effects (51). Together these studies suggest that most of the changes in gene expression are a consequence of knockdown of the intended target, but that each siRNA also nonspecifically perturbs the expression of a limited number of additional transcripts.

An important question is whether this level of nonspecific perturbation of gene expression is sufficient to produce changes in cell physiology. This appears to be the case. For example, some siRNAs targeting the gene MEN1 induced changes in expression of p53 and p21 at the mRNA and protein levels in multiple cell lines (52). However, some siRNAs that knocked down MEN1 equally well did not cause changes in p53 or p21 levels, suggesting that alteration of p53 and p21 expression was a result of an off-target effect. This study illustrates why it is essential to correlate a phenotype with the degree of knockdown of the intended target; a lack of correlation strongly suggests that an off-target effect may be involved.

These studies indicate that the ability of an siRNA to produce an off-target effect depends not only on dose of the siRNA but also on the sequence of the specific siRNA used in the study. Based on the ability of an siRNA to induce microRNA-like knockdown through interactions with the seed region of the siRNA, Anderson et al. (53) hypothesized that siRNAs that contain a higher number of seed matches with 3′UTRs in the genome would show a higher frequency of off-target effects. For each siRNA, they calculated a seed complement frequency (SCF), which represents the number of 3′ UTRs in the genome that contain a sequence match to the seed sequence of the siRNA. They first used gene expression profiling to analyze the effects of knockdown of two housekeeping genes by ten different siRNAs each. These siRNAs had a range of SCFs, ranging from high (>3800), to medium (2500–2800) to low (<350). Transcriptional profiling revealed extensive microRNA-like off-target effects for siRNAs with high or medium SCF, but not for siRNAs with low SCF. In addition, the authors demonstrated that siRNAs with low SCF were three-fold less likely to nonspecifically inhibit viability or induce apoptosis via off-target effects. Together these findings indicate the importance of minimizing the SCF when designing siRNAs to study gene function or in generating siRNA libraries for high-throughput screening.

Properties of the mRNA may also impact its susceptibility to off-target effects. Given the similarity in mechanism between seed-match-based off-target effects and microRNA-based targeting, properties of an mRNA that render it sensitive to microRNA targeting may also increase its susceptibility to off-target effects. Grimson et al. performed an extensive analysis of the determinants that contribute to effectiveness of miRNA targeting in addition to seed sequence pairing (54). Target site location in the 3′UTR, their multiplicity and spacing, and additional pairing of the miRNA nucleotides 12-17 were correlated with enhanced miRNA targeting efficiency. The sequence and structural context of mRNA 3′UTR target sites may also play an important role as it was found that higher local AU-richness, potentially weakening secondary structure to increase target site accessibility, was associated with more effective microRNA-based targeting. It is reasonable to expect that such properties would also enhance susceptibility to microRNA-like off-target effects in siRNA library screens.

Although the majority of seed-match-based off-target effects occur by targeting sites in the 3′UTR, recent cross-linking studies suggest the RISC complexes are targeted to coding regions at a surprisingly high frequency. Cross-linking immunoprecipitation (CLIP) of Argonaute and related RNAi proteins has allowed unbiased identification of RISC target sites in a transcriptome-wide manner (55-57). These studies revealed that although most (66-84%) RISC-binding sites occur in exons, only 46-60% of these occurred in 3′ untranslated regions while 38-50% occurred in coding sequences. Only a small fraction (2-4%) mapped to 5′UTR regions. Interestingly, a significant number of RISC binding sites (12-14%) mapped to pre-mRNA intronic regions. The functional significance of RISC targeting to coding sequences or intronic sequences remains unclear, but Hafner et al.(56) determined that targeting to coding regions caused limited but significant destabilization of corresponding mRNAs. These studies indicate that much remains to be elucidated about the function of small RNA molecules, and therefore additional mechanisms of off-target effects, beyond those that target the 3′ UTR, remain possible.

The impact of off-target effects in RNAi screening

Although off-target effects can occur, the degree of gene modulation is typically two-fold or less. An important practical question is whether such off-target effects are sufficient to produce false-positive hits in high-throughput screens, where it might be expected that a greater degree of knockdown of the intended target would be necessary to produce a strong phenotype. However, several investigators have reported that some of the top hits in their screens arise due to microRNA-based off target effects. In a screen to identify novel regulators of the HIF1-α transcription pathway, the top scoring siRNAs did not act through specific knockdown of their intended targets, but rather by targeting the HIF1- α mRNA through microRNA-like seed match effects against sequences in the 3′ UTR (58). In a separate screen for modulators of sensitivity to the Bcl-2 targeting drug ABT-737, the same group determined that many active siRNAs nonspecifically targeted the key anti-apoptotic protein Mcl-1, again through microRNA-like seed match effects (59). Off-target effects through a microRNA-like mechanism were also prominent in an siRNA screen for novel regulators of the TRAIL apoptosis induction pathway (60), suggesting that RNAi screening enriches for siRNAs with relevant off-target effects. Finally, in a screen to identify new components of the spindle checkpoint pathway, we found that the vast majority of active siRNAs acted by targeting the 3′UTR of the known spindle checkpoint gene Mad2 through seed match effects (F. Sigoillot and R. King, unpublished data). Together these results indicate that when evaluating the results from high-throughput screens, investigators should ensure that active siRNAs targeting novel genes do not contain potential seed matches against known components of the pathway.

The role of siRNA multiplicity in siRNA screening

One commonly practiced method to reduce the impact of off-target effects in high-throughput screens is to screen multiple siRNAs or shRNAs per gene, either as a pool or individual reagents, and then to characterize only those genes for which multiple siRNAs yield a phenotype. In this approach, the chance of the phenotype resulting from an off-target effect is believed to be minimized because each siRNA should have a distinct spectrum of off-target effects. Here we examine the effectiveness of this approach in distinguishing true hits from false positive results in siRNA screens.

Table 1 summarizes the results of twenty-nine siRNA or shRNA screens performed in mammalian cells (Table 1 is adapted and expanded from that of (24)). Analysis of Table 1 indicates that the primary hit rate for these screens varies widely, from a low of 0.4% to a high of 25.6%, with a median of 2.3%. For a genome-wide screen involving 20,000 genes, this would yield approximately 460 primary hits. For two of the published screens, there is no validation of the reported hits from the screens. For fourteen of the screens, validation is provided only by showing that at least two or more siRNAs targeting a given gene produce the same phenotype. For the remaining ten screens, additional validation of hits is performed, but typically only a very small fraction of the total hits are validated by independent methods (ranging from 1–5 genes per screen). Therefore, the overall rate of validation relative to the total hit rate is low. The low rate of hit validation may be explained in part by the fact that some validation approaches may be challenging and time-consuming, as described below. Alternatively, it is possible that even genes that score on multiple siRNAs may still represent an off-target effect, thereby reducing the validation rate. In practice, many groups consider a hit to be a true positive if at least two independent siRNAs for a gene produce the phenotype. Here we question the validity of that assumption.

Table 1. Validated genes yield in RNAi high-throughput screens in mammalian cells (adapted from Mohr et al.(24)).

Cell type	Phenotype	Reagent	SiRNA concentration (nM)	Number of genes/mRNA targets tested	Primary screen hits	Primary Hit Rate (%)	Number of RNAi molecules scoring					Validated genesa	References	Year
							1	2	3	4	>4
HeLa P4/R5	Host genes involved in HIV replication	Pooled siRNAs	not provided	19,709	936	4.7			1	9		4	(75)	2008
Jurkat	Host genes involved in HIV replication	Pooled shRNAs	n.a.	54,509 mRNAs/ESTs	252	n.a.	n.a.	n.a.	n.a.	n.a.	n.a.	3	(86)	2009
HeLa	Host genes involved in west Nile Virus infection	Pooled siRNAs	50	21,121	305	1.4		180	97	18		2	(95)	2008
HeLa-derived TZM-b1	Host genes involved in HIV replication	Pooled siRNAs	50	21,121	386	1.8	191		26			2	(84)	2008
HeLa	Spindle assembly checkpoint genes	shRNA	n.a.	780	16	2.1	6		2			1b	(89)	2007
HeLa	Regulators of Wnt/β-Catenin pathway	Pooled siRNAs	not provided	21,125	530	2.5	214		16			1	(108)	2008
293T, HeLa, MCF7	Regulators of genotoxic stress-induced apoptosis	Pooled shRNAs	n.a.	genome-wide	30	n.a.	29		1			1c	(92)	2008
Huh7/Rep-Feo	Host genes required for HCV replication	Pooled siRNAs	50	21,094	236	1.1	148		35			1	(107)	2009
mouse ES	Regulators of mTERT gene transcription	shRNA	n.a.	1,360	18	1.3	17	1				1c	(88)	2010
Huh 7.5.1	Host genes required for HCV replication	Pooled siRNAs	50	19,470	521	2.7		262				5d	(96)	2009
HeLa	Cell cycle progression and division	esiRNA	30	5,305	275	5.2	n.a.	n.a.	n.a.	n.a.	n.a.	0	(93, 94)	2004
H1299	HIF1 alpha signal transduction	siRNA	100	507	--	n.a.	n.a.	n.a.	n.a.	n.a.	n.a.	0e	(58)	2005
U2OS	Cell cycle progression and division	siRNA	12.5	24,373 and 5,000f	1,152	n.a.	505	612				0	(101)	2006
HT29	Regulators of mitosis	shRNA	n.a.	1,028 (4903 shRNAs)	102	n.a.		66	19	2	1	0	(100)	2006
NCI-H1155	Modulators of responsiveness to Taxol	Pooled siRNAs	40	21,121	87	0.4		24				0	(109)	2007
NIH3T3	Modulators of oxidative stress resistance	Pooled shRNAs	n.a.	genome-wide	15	n.a.	14			1		0	(103)	2007
NCI-H196	Modulators of Bcl-2/Bcl-XL inhibitor ABT-737-induced death	Pooled siRNAs	25	4,000	145	3.6	3					0e	(59)	2007
MCF10A	Genes essential for viability of breast cancer cells	Pooled shRNAs	n.a.	1000s	201	n.a.		6	3			0g	(106)	2008
293T	Host genes involved in HIV replication	Pooled siRNAs	n.a.	19,628	4,555	23.2		295				0	(85)	2008
DLD1	Regulators of Wnt/β-Catenin pathway	Pooled siRNAs	25	20,042	740	3.7		119				0	(99)	2008
HEK293	Regulators of cell adhesion	Pooled shRNAs	n.a.	8500 (43,828 shRNAs)	--	n.a.		1				0	(91)	2008
Jurkat	Resistance to FAS-induced apoptosis	Pooled shRNAs	n.a.	9500 (45,000 shRNAs)	11	n.a.		5				0	(97)	2008
MNT-1	factors involved in melanin biogenesis	Pooled siRNAs	1.4	21,127	98	0.5		25				0	(90)	2008
RDG3	Stress granules and Processing bodies assembly	Pooled siRNAs	40	7,317	--	n.a.		171				0	(104)	2008
BJtsLT	biomarkers for response to an anticancer drug	Pooled shRNAs	n.a.	8,000	100	1.3		6				0	(102)	2009
DLD1	Synthetic lethal interactions with Ras oncogene	Pooled shRNAs	n.a.	32,293	368	1.1	357	11				0	(98)	2009
U2OS	Modulators of circadian clock genes transcription	Pooled siRNAs	12.5	22,468	5,751	25.6	5,383	368				0	(110)	2009
HeLa	Modulators of TRAIL-induced apoptosis	siRNA	2.5pmol/96well	6,095	752	12.3	721	31				0	(60)	2010
GL261 derivative	Regulators of ATF5 transcription	Pooled shRNAs	n.a.	∼28,000	12	n.a.		3				0	(105)	2010

^aGene validated by rescue expression or with an independent method (not multiplicity of siRNAs)

Bioinformatics database cross-referencing was not considered as validation since it is not clear how information in the databases is itself validated.

^bTAO1 Kinase siRNA was rescued by expression of a siRNA-resistant form, but other work (80) suggests this may be an off-target effect.

^cNo rescue experiments were performed. assays independent of RNAi were performed (mTERT promoter-luciferase construct were shown to have a functional top hit HIF1alpha consensus site (88); Top hit TAF1 (92) overexpression induced apoptosis).

^d5 hits were previously reported to interact with HCV core component proteins. No rescue experiments were performed.

^eTop hits were indentified as seed-match induced off target effects

^fTwo overlapped siRNA libraries were used

^gThe authors report a pooled shRNA methodology and do not focus on gene-phenotype association, thus no validation is attempted

n.a. not available

In Table 2, we consider a hypothetical genome-wide screen (targeting 20,000 genes) using 4 siRNAs per gene. Each siRNA is screened individually, or as a pool and then deconvoluted, to identify which of the four siRNAs is active in the assay. This yields a list of candidate genes, as well as the number of siRNAs that are active for each gene. Published studies have indicated that each siRNA may nonspecifically reduce the expression of 30–100 genes by two-fold or more due to microRNA-like effects (46-49, 61). Based on this information, we assumed that an average siRNA might downregulate 50 genes by two-fold or more through an off-target microRNA-like effect. We then calculated the expected frequency of on-target and off-target genes that would be discovered in a screen, based on the number of genes in the pathway that would yield a phenotype if downregulated by two-fold or more. For the off-target genes, we calculated the frequency with which a gene would be expected to be targeted by one or more of the four different siRNAs.

Table 2.

Estimation of the number of off-target siRNAs that may arise from genome-wide RNAi screens.

In this analysis, we calculate the hit rate and validation rate in an siRNA screen of 20,000 genes using four siRNAs per gene. Genes chosen for validation would include any gene for which at least 2/4 siRNAs produce a phenotype; we assume that at least 2/4 siRNAs that target the true (specific) genes involved in the pathway would yield a phenotype.

Assuming that a given siRNA has on average 50 off-target mRNAs (leading to mRNA knockdown by more than 2 fold) and that these off-targets are random, the probability that an siRNA has a seed match that results in off-targeting for an mRNA out of 20,000 genes is 50/ 20,000 = 1/ 400. Let Y denote the number of genes among the indicated number of genes required for phenotype n ={1, 5, 10, 20, 50} that have a seed match to a given siRNA. If each of the n genes is independent of each other, then P(Y) obeys the binomial distribution B (n, 1/400). Therefore the probability of a given siRNA not to contain a seed match to any of the n genes is given as follows:

P(Y=0) = C_10,0 (1/400)⁰ (1/400)^n-0

The probability that a given siRNA has a seed match to at least one of the n genes is

p = 1 - P(Y=0)

The probability P that exactly A = {1, 2, 3, 4} siRNAs out of 4 independent siRNAs tested for a given gene give phenotype as a result of an off-target effect follows the binomial distribution:

P(A) = C_4,A p^A p^4-A

The estimated number of genes with exactly A out of 4 siRNAs giving phenotype as a result of an off-target effect was determined by multiplying the probability P(A) for each n by 20,000, assuming this number of genes was tested in the siRNA screen.

Number of Genes in a Pathway that Yield Phenotype when Expression Reduced by 50%	1	5	10	20	50
A. Number of On-target Genes That Score	1	5	10	20	50
B. Number of Off-target Genes That Score	200	976	1906	3630	7877
B1. Number of genes with exactly 1/4 siRNAs that score	199	958	1835	3362	6465
B2. Number of genes with exactly 2/4 siRNAs that score	1	18	70	259	1293
B3. Number of genes with exactly 3/4 siRNAs that score	0	0	1	9	115
B4. Number of genes with exactly 4/4 siRNAs that score	0	0	0	0	4
Total Number of Genes Identified by Screen (A+B)	201	981	1916	3650	7927
Hit Rate ([A+B]/20,000)	1%	4.9%	9.5%	18%	39%
Fraction of Genes that are On Target (A/[A+B])	0.5%	0.5%	0.5%	0.5%	0.6%
Validation Rate (A/[A+B2+B3+B4])	50%	21.7%	12.3%	6.9%	3.4%

Inspection of Table 2 reveals several points. First, regardless of the number of genes involved in the pathway, the vast majority of hits are due to off-target effects. Second, the usefulness of siRNA multiplicity for distinguishing between on-target and off-target effects depends strongly on the number of genes that participate in the pathway. For example, if only one gene is involved in a pathway, then it is reasonably likely that any gene for which two of four siRNAs score is due to knockdown of the intended gene rather than to an off-target effect. However, if there are ten or more genes involved in a pathway, the majority of genes that confirm on two out of four siRNAs are instead likely to result from an off-target effect. For example, with ten genes in a pathway, eighty-one genes would score on two or more siRNAs, yet only ten of these genes would represent a specific result of knockdown of the intended gene. The problem becomes worse as the number of genes involved in a pathway increases. Finally, it is also evident that the overall hit rate can become very high, even if there are only 20 genes required for the phenotype. It is likely that many cellular pathways rely on even larger numbers of genes. Based on the likelihood of such off-target effects, it is not surprising that several of the published screens show primary hit rates in excess of twenty percent, with a very low rate of validation of hits.

Approaches for validation

Several reviews (24, 36, 62-64) and editorials (65) have discussed approaches for validating the results of siRNA screens, but as can be seen in Table 1, most hits from siRNA screens are not validated. As we have discussed, caution must be used when employing a criterion of multiple siRNAs to validate a result, especially if the pathway being studied consists of many genes that could contribute to the phenotype. If multiple siRNAs are used to validate a result, it is essential that they be distinct from the siRNAs used in the original screen, because the siRNAs from the screen have been subject to selection by the screening process.

Approaches involving chemical or structural modifications of siRNAs have been developed in an attempt to reduce the likelihood of off-target effects (reviewed in(66)). These modified siRNAs provide an additional tool to validate that knockdown of the intended mRNA is responsible for the observed phenotype. One approach is to modify the passenger strand of the siRNA, to reduce the likelihood that it would be incorporated into RISC, and thereby eliminate any off-target effects due to seed-match effects caused by the passenger strand. Different methods have been developed to neutralize passenger strand incorporation into RISC, including design of asymmetric siRNAs with a shorter 15 nucleotide passenger strand (67) or an internally segmented passenger strand (68). In both cases off-target effects from the passenger strand were reduced while guide strand RNAi efficiency was maintained. Another approach is to chemically modify all nucleotides of the passenger strand with 2′-O-methyl substitution (69). This modification reduces passenger strand incorporation while preserving guide strand incorporation.

Modifications of the guide strand have been developed to reduce the likelihood of seed-match-based off-target effects. One approach is to reduce the thermodynamic stability of the interaction of the seed region with the target mRNA. This can be accomplished by substituting the 8 base pair region at the 5′-end of the guide strand with double strand DNA (70), or by using unlinked nucleic acid (in which the carbon-carbon bond between the 2′ and 3′ of the ribose sugar is broken) at the 7 position of the seed region (71). Using reporter assays, both of these methods have been shown to reduce microRNA-like off-target effects, although the approach has not yet been validated across a range of different siRNAs that target endogenous mRNAs. Alternatively, introduction of 2′-O-methyl substitution at the second position of the seed region has been proposed to alter the manner in which RISC interacts with target mRNAs (72), selectively reducing microRNA-like off-target effects while leaving on-target silencing intact. Jackson et al. (72) found that 2′-O-methyl substitution of the guide strand at positions one and two of the seed region reduced off-target silencing without inducing novel off-targets (72). Out of 10 modified siRNAs tested, all retained strong silencing of the intended target while reducing silencing of 80% of off-target mRNAs by an average of 66%, although the effectiveness varied widely among different mRNAs. Jackson et al. determined (72) that siRNAs with stronger hybridization in the seed region (lower free energy of RNA:RNA binding for bases 2-7) responded less efficiently to 2′-O-methyl modifications than weaker-hybridizing siRNAs, indicating that the modification does not fully abrogate all off-target effects. Because siRNAs with strong seed hybridization may be more effective in reducing expression, they may produce strong phenotypes and may be more likely to be identified in siRNA screens. Therefore, confirmation of a result using the 2′ O-methyl modified version of an siRNA discovered in a screen cannot rule out the possibility that the phenotype is due to an off-target effect. Nevertheless, use of these modified reagents in high-throughput screens may reduce the rate of false-positive hits (73).

Once multiple siRNAs against an intended target have been tested and some shown to induce studied phenotype, a simple approach that can help validate whether an effect is specific is to determine whether the level of knockdown of the intended gene, as measured by protein expression, correlates with phenotype across the series of siRNAs. If siRNAs are identified that cause efficient knockdown of the protein target, but do not induce phenotype, then it is almost certain that an off-target effect contributes to the phenotype.

Rescue experiments represent the gold standard for validation of siRNA screening results. However, as shown in Table 1 most published screens do not include such an approach. Rescue experiments test whether expression of a non-targetable form of the gene is sufficient to rescue the phenotype produced by an siRNA or shRNA. The rescue construct may contain engineered silent mutations that abrogate complementarity to the siRNA, or it may be an evolutionarily diverged form of the gene, such as a mouse gene used to rescue an siRNA-induced phenotype in human cells (74). Alternatively, if siRNAs targeting the 3′ UTR of a gene are used, rescue experiments can be performed with cDNA constructs that lack the 3′ UTR (75). Even when focusing on genes validated using multiple siRNAs, the rate of rescue of phenotype may be lower than expected, consistent with a high frequency of off-target effects even among genes scoring on multiple siRNAs. For example, Zhou et al. performed rescue experiments for nine different genes that were validated by multiple siRNAs in their screen; however, in only four of nine cases could the phenotype be rescued by expression of a non-targetable cDNA (75).

There are several potential pitfalls in designing and executing successful rescue experiments. One issue is that it can be challenging to express the rescue construct at physiological levels. If the phenotype is highly sensitive to gene dose, the experiment can be challenging to interpret. This is a major advantage of using mouse bacterial artificial chromosome (BAC) constructs for rescuing phenotypes in human cells (74, 76), as the protein is more likely to be expressed at physiological levels. Another issue is that it is essential to confirm that knockdown of the endogenous protein is not hampered by expression of the rescue construct. If the rescue construct has even limited complementarity with the siRNA, it can act as a sponge to reduce the effective concentration of the siRNA (77-79). This problem can be controlled for by ensuring that knockdown of the endogenous gene is not reduced when the transgene is expressed, or avoided entirely by a rescue strategy in which cDNAs that lack the 3′UTR are used to rescue the phenotype of siRNAs that target the 3′ UTR of the endogenous gene. Finally, there is a reported case of a false-positive rescue experiment, in which overexpression of the targeted gene rescued the phenotype by sequestration of another regulator of the pathway, reversing an off-target effect (80). Ultimately, the use of methods independent of RNAi machinery should be considered to fully validate the involvement of selected genes in the studied phenotype. For example one can conditionally knockout the genes of interest (81) or test small molecule inhibitors if they are available (82, 83).

Factors influencing the false-negative rate in RNAi-based screens

In designing an RNAi screen, another important issue to consider is how different approaches affect the rate of false-negative results. Although it is difficult to directly compare different screens due to methodological differences or cell types used, four published screens to identify host factors required for HIV replication identified no hits in common across all four screens (75, 84-86). Comparison of each of these screens to each other yielded a higher overlap, from nine to eighteen genes. However, given the fact that each screen identified hundreds (75, 84) or thousands of potential hits (85), the low degree of overlap is surprising. There are several potential explanations for the lack of overlap. First, methodological differences, including the particular cell line that is used, could explain the lack of overlap. Second, as discussed earlier, if the proportion of off-target effects is high, and these effects are dependent on the library or cell line used, a low overlap of genes would be expected. Finally, the screens may simply not be saturating, with a high rate of false-negative results in each screen. This could be explained by differences in the ability of different siRNAs or shRNAs to knockdown the gene of interest. This may depend on characteristics of the siRNA as well as characteristics of the targeted mRNA and protein. Indeed, some proteins with a half-life in days may not be decreased sufficiently when the phenotype is assessed, leading to a false negative result. In theory, it may be possible to achieve more profound knockdown of protein levels using stable expression of shRNAs under conditions of selection, as long as the gene is not essential for cell viability. There may be some genes that are difficult to target with siRNA or shRNA approaches. For example, it has been shown that more abundant mRNAs may tend to dilute out siRNAs, thereby decreasing the efficiency of knockdown (87). In addition, in many cases only a small amount of a target protein may be necessary for adequate function in the cell. Finally, functional redundancy may hamper the identification of genes when evaluating the consequence of knockdown one gene at a time. In this case, a phenotype may be revealed only after simultaneous knockdown of multiple members of a gene family.

Concluding thoughts

RNAi-based approaches have shown tremendous potential for helping us understand gene function in individual experiments. More recently, the advent of genome-wide RNAi-based screens has helped identify new components of various pathways. However, analysis of published screens indicates that the rate of identification of validated genes is less than what might have been expected. Here we have reviewed some potential pitfalls of RNAi-based screening that arise from the potential of off-target effects. Although siRNA and shRNA library design is improving to reduce the magnitude of these effects, it is essential that investigators remain vigilant about the possibility of off-target effects, and design screens and follow-up strategies to validate individual genes to ensure that the screen yields useful insights. Screens against pathways or cellular processes that involve large numbers of different components (cell proliferation, cell death, cell signaling) may be especially prone to a high rate of false positive results. In the future, performing parallel screens with two independent siRNA libraries may help reduce the rate of both false positive and false negative results. Validation of results using independent siRNA reagents as well as rescue experiments is essential to have confidence in the role of a particular gene in the process. Investigators must have an attitude of skepticism toward the role of a novel gene in a process until it has been independently validated. Only under these circumstances will RNAi-based screening meet its tremendous potential as a discovery tool.