Selection of reference genes for quantitative PCR: identifying reference genes for airway epithelial cells exposed to Pseudomonas aeruginosa

Abstract

Most quantitative PCR (qPCR) experiments report differential expression relative to the expression of one or more reference genes. Therefore, when experimental conditions alter reference gene expression, qPCR results may be compromised. Little is known about the magnitude of this problem in practice. We found that reference gene responses are common and hard to predict and that their stability should be demonstrated in each experiment. Our reanalysis of 15 airway epithelia microarray data sets retrieved from the National Center for Biotechnology Information (NCBI) identified no common reference gene that was reliable in all 15 studies. Reanalysis of published RNA sequencing (RNA-seq) data in which human bronchial epithelial cells (HBEC) were exposed to Pseudomonas aeruginosa revealed that minor experimental details, including bacterial strain, may alter reference gene responses. Direct measurement of 32 TaqMan reference genes in primary cultures of HBEC exposed to P. aeruginosa (strain PA14) demonstrated that choosing an unstable reference gene could make it impossible to observe statistically significant changes in IL8 gene expression. We found that reference gene instability is a general phenomenon and not limited to studies of airway epithelial cells. In a diverse compendium of 986 human microarray experiments retrieved from the NCBI, reference genes were differentially expressed in 42% of studies. Experimentally induced changes in reference gene expression ranged from 21% to 212%. These results highlight the importance of identifying adequate reference genes for each experimental system and documenting their response to treatment in each experiment. This will enhance experimental rigor and reproducibility in qPCR studies.

INTRODUCTION

Many excellent reviews offer advice on how to perform quantitative PCR (qPCR) correctly (4, 13–15, 23–25, 34, 50). Most of these reviews highlight the importance of choosing a reference gene carefully and, specifically, recommend against using a popular reference gene, such as β-actin (ACTB) or glyceraldehyde-3-phosphate dehydrogenase (GAPDH), without validating its stability in a new experimental context. This advice is based partly on reports that common reference genes may be expressed at different levels in different tissues, sometimes varying by >10-fold between tissues (52) and >30-fold in blood (15). Although uncommon, reference gene changes of this magnitude could clearly overwhelm typically reported changes in genes of interest. However, because a systematic analysis of qPCR reference gene responses to treatments in typical experiments has not been published, it is difficult to estimate the potential impact of reference gene instability on qPCR results.

Many approaches have been suggested to reduce the likelihood of using an unstable reference gene in qPCR: algorithms have been developed to identify the most stable reference gene from a series of candidates (13, 19, 34, 38, 45). Tools have been developed to identify candidate reference genes based on gene expression (8, 10, 12, 29, 39, 40), and standards have been developed to document every aspect of qPCR methodology, including the process by which a reference gene was selected (6). Nonetheless, there is scant evidence that the research community has paid much attention to the problem of reference gene instability or its solutions. GAPDH and ACTB are still used as references in most published studies, and only 15% of studies report validation of the reference that is used (9).

Lack of compliance with recommended qPCR methodology may compromise the quality of results, but the magnitude of this problem is largely unknown. In a recent publication, Stephen Bustin, a leader in the qPCR community (5–7, 15–17, 30, 43), stated that the majority of qPCR results are “technical noise” (5). This estimate was based on a review of the methods used in 20 published studies and a careful analysis of the various biases in qPCR, including use of an unstable reference gene, which can influence results. In this report, we have taken a data-driven approach to assess the impact of reference gene instability on qPCR conclusions.

We used four approaches to examine how the selection of a reference gene might affect conclusions in qPCR studies. 1) We used ScanGEO (36), an online gene expression data-mining tool, to assess whether 10 commonly used reference genes (Table 1) were stably expressed in 15 publicly available studies of human airway cells. 2) We reanalyzed an existing RNA sequencing (RNA-seq) data set in which airway epithelial cells were exposed to Pseudomonas aeruginosa. 3) We used a TaqMan reference panel of 32 genes to assess the effect of P. aeruginosa on reference gene expression in primary human airway epithelial cells. 4) We analyzed 986 publicly available human gene expression studies to identify the frequency and magnitude of differential gene expression of 32 reference genes and other genes. Based on these analyses, we conclude that reference gene expression is highly dependent on experimental conditions and that reference genes are frequently differentially expressed in response to treatment or between different experimental groups. Our analysis highlights the importance of explicit documentation of the behavior of reference genes in all publications.

Table 1.

Ten common reference genes (52) interrogated using ScanGEO (36)

Gene Symbol	Description
ACTB	β-Actin
B2M	β2-Microglobulin
GAPDH	Glyceraldehyde-3-phosphate dehydrogenase
HMBS	Hydroxymethylbilane synthase
HPRT1	Hypoxanthine phosphoribosyltransferase 1
RPL13A	Ribosomal protein L13a
SDHA	Succinate dehydrogenase complex flavoprotein subunit A
TBP	TATA-box binding protein
UBC	Ubiquitin C
YWHAZ	Tyrosine 3-monooxygenase/tryptophan 5-monooxygenase activation protein-ζ

METHODS

Meta-analysis of reference genes in human airway epithelial cell microarray data.

We used ScanGEO (http://scangeo.dartmouth.edu/ScanGEO/) to conduct a meta-analysis of reference genes in 15 human airway epithelial cell microarray data sets. ScanGEO is a publicly available Shiny web application that can be used to identify differentially expressed genes, visualize results, and provide summary statistics (36). We identified data sets in ScanGEO by selecting genus (Homo) from a pulldown list and entering “airway epithelia” into a text box. Next, we uploaded the 10 commonly used reference genes (52) listed in Table 1 from a comma-separated value file. Then, using a significance threshold of P < 0.05, the default setting (36), we analyzed the 15 identified studies for differential expression of the 10 reference genes. Results were retrieved in a zipped file. The studies examined and their Gene Expression Data Set (GDS) annotation are presented in Table 2.

Table 2.

15 Microarray studies referring to “airway epithelia”

Title	Gds
Large airway epithelial cells from cigarette smokers with suspect lung cancer	GDS2771
Staphylococcal α-toxin effect on airway epithelial cells	GDS5812
Airway epithelial cell response to various cigarette smoke condensates	GDS4917
Asthmatic atopic epithelium	GDS3711
Small airway epithelium response to cigarette smoking	GDS2486
Large airway epithelium response to cigarette smoking (HuGeneFL)	GDS2489
Cystic fibrosis transmembrane conductance regulator expression in airway epithelial cells	GDS4255
Airway epithelial cell response to IL-13 in vitro	GDS4981
Airway epithelial cell response to Pseudomonas aeruginosa rsmA mutant infection	GDS2287
Large airway epithelium response to cigarette smoking (HG-133A)	GDS2490
Large airway epithelium response to cigarette smoking (HG-U133 2.0)	GDS2491
Airway epithelium response to injury: time course	GDS2495
Mucociliary differentiation of airway epithelial cells	GDS2615
Airway epithelial cell response to hypochlorous acid: time course	GDS3363
Bronchial epithelial cell response to rhinovirus infection and cigarette smoke exposure	GDS4832

Meta-analysis of reference genes and other Kyoto Encyclopedia of Genes and Genomes genes in microarray data.

To examine reference gene expression in a larger data set of microarray studies on human airway epithelial cells, a compendium of 1,775 studies was constructed using GEOmetadb (54) and the database query “select gds from gds where platform_organism like '%HOMO%' ” in the category “expression profiling by array.” For each study, we queried expression values for 7,148 Kyoto Encyclopedia of Genes and Genomes (KEGG) genes (32, 33), which included the 10 reference genes listed in Table 1. Studies with missing values for more than half of the genes were eliminated, leaving 986 studies. KEGG genes detected in fewer than half of all studies were eliminated, leaving 6,356 KEGG genes. Titles and GDS numbers for the 986 studies are available in Supplemental Table S1 (see https://doi.org/10.6084/m9.figshare.12140961.v1). We downloaded normalized series matrix files from Gene Expression Omnibus (GEO) to perform further analysis, because raw data are not always available and we did not want to introduce systematic effects by normalizing the data in some cases but not others. Normalized data were log-transformed as needed. ANOVA P values and maximum absolute log₂ fold changes were calculated in R (46a) for each gene observed in each study, yielding 5,845,935 observations of P values and maximum fold change. Wilcoxon’s rank sum tests in R were used to assess significant differences between the numbers of times different gene classes achieved significance.

Analysis of reference genes in P. aeruginosa-exposed human airway epithelial cells by RNA-seq.

Our laboratory previously published work in which airway epithelial cells were exposed to P. aeruginosa for 6 h (21) and gene expression was assessed by RNA-seq. Using only data from samples that were not exposed to arsenic and modeling donor effects as well as exposure to P. aeruginosa, we reanalyzed these data by downloading count tables and determining differential expression in edgeR (47). Output from edgeR was mapped to gene symbols derived from 32 reference genes in Table 3, yielding observations of ACTB, GAPDH, and other common reference genes.

Table 3.

TaqMan Array human endogenous controls

TaqMan Probe	Description
18S-Hs99999901_s1	18S ribosome
ABL1-Hs00245445_m1	ABL proto-oncogene 1, nonreceptor tyrosine kinase
ACTB-Hs99999902_m1	β-Actin
B2M-Hs99999907_m1	β2-Microglobulin
CASC-Hs00201226_m1	CASC3 exon junction complex subunit
CDKN1A-Hs00355782_m1	Cyclin-dependent kinase inhibitor 1A
CDKN1B-Hs00153277_m1	Cyclin-dependent kinase inhibitor 1B
EIF2B-Hs00426752_m1	Eukaryotic translation initiation factor 2B subunit α
ELF1-Hs00152844_m1	E74-like ETS transcription factor 1
GADD45A-Hs00169255_m1	Growth arrest and DNA damage inducible-α
GADPH-Hs99999905_m1	Glyceraldehyde-3-phosphate dehydrogenase
GUSB-Hs99999908_m1	β-Glucuronidase
HMBS-Hs00609297_m1	Hydroxymethylbilane synthase
HPRT1-Hs99999909_m1	Hypoxanthine phosphoribosyltransferase 1
IPO8-Hs000183533_m1	Importin 8
MRPL19-Hs00608519_m1	Mitochondrial ribosomal protein L19
MT-ATP6-Hs02596862_g1	Uncharacterized LOC101928524
PES-Hs00362795_g1	hCG2010949 Celera annotation; pescadillo ribosomal biogenesis factor 1
PGK-Hs99999906_m1	Phosphoglycerate kinase 1
POLR2A-Hs00172187_m1	RNA polymerase II subunit A
POP4-Hs00198357_m1	POP4 homolog, ribonuclease P/MRP subunit
PPIA-Hs99999904_m1	Peptipylprolyl isomerase A
PSMC4-Hs00197826_m1	Proteasome 26S subunit, ATPase 4
PUM1-Hs00206469_m1	Pumilio RNA binding family member 1
RPL30-Hs00265497_m1	Ribosomal protein L30
RPL37a-Hs01102345_m1	Ribosomal protein L37a
RPLP0-Hs99999902_m1	Ribosomal protein lateral stalk subunit P0
RPS17-Hs00734303_g1	Ribosomal protein S17
TBP-Hs99999910_m1	TATA-box binding protein
TFRC-Hs99999911_m1	Transferrin receptor
UBC-Hs00824723_m1	Ubiquitin C

Exposure of human bronchial epithelial cells to P. aeruginosa.

Primary human bronchial epithelial cells (HBEC) from three donors obtained from Dr. Scott Randell (University of North Carolina, Chapel Hill, NC) were cultured in BronchiaLife basal medium (Lifeline Cell Technology, Frederick, MD) supplemented with the BronchiaLife B/T LifeFactors Kit (Lifeline) as well as 10,000 U/ml penicillin and 10,000 μg/ml streptomycin, as described previously (20). The Dartmouth Committee for the Protection of Human Subjects has determined that the use of HBEC in this study is not considered human subject research, because cells are taken from discarded tissue and contain no patient identifiers.

At passage 5, HBEC from three donors were exposed to overnight cultures of P. aeruginosa strain PA14 grown in lysogeny broth (LB, Invitrogen, Grand Island, NY) for 1 h at multiplicity of infection of 10 in the absence of antibiotics. HBEC were washed twice with medium containing 75 µg/ml gentamicin and then incubated for 24 h with 75 µg/ml gentamicin present in the culture medium before RNA isolation. Unexposed control cells from each donor were treated in a similar manner.

qPCR measurements.

At 24 h after PA14 exposure, HBEC were washed with PBS, and RNA was isolated using the miRNeasy kit (Qiagen). Four micrograms of RNA were used as input for cDNA synthesis with SuperScript IV (Invitrogen, Grand Island, NY). qPCR for IL8 (our gene of interest) and ubiquitin C (UBC, a common reference gene) was performed using the standard protocol and cycle conditions for the TaqMan Fast Advanced Master Mix (Thermo Fisher Scientific).

Cycle threshold (C_t) values for IL8 and UBC were analyzed using linear models in R. A two-factor model of C_t response to P. aeruginosa [yes/no (Y/N)] and donor (25J, 42I, or 7E) was used to estimate the responsiveness of IL8 and UBC.

TaqMan Array Human Endogenous Control plates (Thermo Fisher Scientific) were run according to the manufacturer’s instructions. Each plate measures the 32 genes described in Table 3. Triplicate measurements of each of the 32 genes on each of the six plates were averaged. For each plate, the difference between each of the 32 genes and the 31 other genes was calculated. Paired comparisons were made for airway cells from each of three donors (25J, 42I, and 7E), which were exposed to P. aeruginosa or left unexposed. Treatment responses of raw C_t values for each of the 32 genes were analyzed for response to P. aeruginosa using two-factor linear models (donor = 25J/42I/7E, exposed = Y/N), yielding estimates of the effect of P. aeruginosa exposure in C_t units and the probability that the effect was due to chance. Finally, means of raw triplicate measurements on the six TaqMan plates were assessed for stability using the selectHKs function in the NormqPCR R package (45), yielding the three most-stable reference genes, hydroxymethylbilane synthase (HMBS), UBC, and ribosomal protein L30 (RPL30). The geometric mean of these three genes was calculated to yield a composite reference signal.

RESULTS

Common reference genes were often differentially expressed in airway epithelial cell microarray studies.

Using the online tool ScanGEO, we performed a meta-analysis of 10 commonly used reference genes (Table 1) from 15 human microarray studies on airway epithelial cells (Table 2). In one airway study (GDS2771, “large airway epithelial cells from cigarette smokers with suspect lung cancer”), all 10 of the common reference genes were differentially expressed between experimental groups (Fig. 1, far left). However, in 4 of 15 studies, none of the 10 common reference genes was differentially expressed (Fig. 1, far right). In other words, reference gene stability in one study on airway epithelial cells does not predict stability in others. No single gene performed well across all 10 studies. β₂-Microglobulin (B2M) was the most often differentially expressed reference gene between experimental groups (ANOVA P < 0.05) in 6 of 15 studies, followed by GAPDH, a commonly used reference gene (Fig. 2).

Fig. 1.

Number of differentially expressed (ANOVA P < 0.05) common reference genes in each of 15 human airway studies and their respective Gene Expression Omnibus Data Series (GDS) numbers (Table 2). Studies are expected to reach P < 0.05 for ~5% of genes by chance alone, as indicated by the horizontal dashed line.

Fig. 2.

Number of times each of 10 common reference genes (Table 1) achieved statistical significance in 15 human airway microarray studies (Table 2). Genes are expected to reach P < 0.05 for ~5% of the studies by chance alone, as indicated by the horizontal dashed line.

Since much of our work specifically involves airway epithelial cells exposed to P. aeruginosa (1–3, 21, 26–28, 31, 35, 41, 48, 49), we analyzed GDS2287 (“airway epithelial cell response to P. aeruginosa rsmA mutant infection”) (44) in greater detail, mapping genes from Table 3 to human genes using ScanGEO. Three reference genes, B2M, HMBS, and peptipylprolyl isomerase A (PPIA), were differentially expressed in response to P. aeruginosa, but the responses appear to be strain-dependent, as shown in Fig. 3. For example, B2M was modestly induced [0.37 units on a log₂ scale, a ∼30% increase (ANOVA P < 0.05)] by exposure to wild-type P. aeruginosa (PAO1). Induction of B2M by infection is consistent with its recently identified role as the precursor to an antibacterial peptide (11), and the fact that it is regulated by PAO1, but not the rsmA mutant, is generally consistent with the findings of Chiou et al. (11). They hypothesized that strain differences in P. aeruginosa affect certain genes of interest in the host, e.g., Kruppel-like factors. Here, we report that strain differences also affect reference gene expression.

Fig. 3.

Expression values of 3 reference genes from Table 3 that reached significance (ANOVA P < 0.05) in GDS2287 (Table 2) in which airway cells were exposed to Pseudomonas aeruginosa (PAO1, ▲) or a mutant strain (rsmA, ■) or unexposed (●). Since these are log₂ values, a difference of 1 between any two groups would indicate an absolute fold change of 2.

Reference gene stability in airway cells exposed to P. aeruginosa is unpredictable.

Figures 1–3 demonstrate the difficulty of identifying a single reference gene for use in all qPCR studies involving airway epithelial cells. In fact, our analysis suggests that a single reliable reference gene for all experiments in which airway cells are exposed to bacterial pathogens might be hard to find: even different strains of P. aeruginosa seem to drive different responses in reference genes. We followed up on this by comparing reference gene behavior in data from a recent report of RNA-seq in which airway epithelial cells were exposed to P. aeruginosa (21) with data from the microarray study by O’Grady et al. (44). Although both studies exposed airway epithelial cells to P. aeruginosa, the experimental design was not identical, as summarized in Table 4.

Table 4.

Details of experimental designs that produced gene expression values in Fig. 3 and responses in Fig. 4

	O’Grady et al.	Goodale et al.
Lung epithelial cell	Immortalized cell line	Primary cells
CFTR genotype	∆F508	Wild type
Culture condition	Plastic	Polarized on filters
Pseudomonas strain	PAO1, rsmA mutant	PA14
Exposure duration	4 h	6 h
RNA isolated at	4 h after infection	24 h after infection
Measurement	Microarray	RNA-seq

Experimental design of O’Grady et al. (44) produced values in Fig. 3, and experimental design of Goodale et al. (21) produced responses in Fig. 4. RNA-seq, RNA sequencing.

Figure 4 shows the P. aeruginosa response of the 30 reference genes in Table 3 that could be mapped to gene symbols in the RNA-seq data of Goodale et al. (21). Genes at the top of Fig. 4 were significantly induced by P. aeruginosa, genes at the bottom were significantly repressed, and genes in the middle were neither induced nor repressed. B2M and GAPDH, which were frequently differentially expressed in 15 airway studies (Fig. 2), were among the most stable in the report of Goodale et al. Importantly, these data suggest that B2M might be a good reference for airway cells exposed to PA14, even though it was differentially expressed in airway cells exposed to PA01 (Fig. 3). Other differences in the experimental design shown in Table 4 could also explain differences in reference gene response to P. aeruginosa.

Fig. 4.

Estimated response of reference genes from Table 3 that could be mapped to an RNA sequencing data set in which airway cells were exposed to Pseudomonas aeruginosa (PA14) for 6 h (21). Response to exposure estimated by edgeR is shown on x-axis. Gray bars indicate significant response (P < 0.5). False indicates P ≥ 0.05. True indicates P < 0.05.

Responses in HBEC reference genes could alter the perceived differential expression of a gene of interest such as IL8.

Next, we considered how the use of different reference genes might affect relative measurements in genes of interest. The TaqMan Array Human Endogenous Control panel was used to measure C_t values for 32 candidate reference genes in each of six samples. Exposure conditions were identical to those used in the study of Goodale et al. (21), as shown in Table 4, except different donors for HBEC were used and cells were exposed to PA14 for 1 h instead of 6 h. IL8 C_t values from the same samples were measured using qPCR and normalized to each of the 32 TaqMan measurements in the same sample, enabling us to use linear models to estimate the effect of P. aeruginosa exposure on normalized IL8 gene expression for each candidate reference gene. In P. aeruginosa, about half (18 of 32) of the reference genes elicited a significant increase in IL8, as shown in Fig. 5. The choice of reference gene also affected calculations of IL8 responses, which ranged from a −1.24 [E74-like ETS transcription factor 1 (ELF1)] to a −0.352 [phosphoglycerate kinase 1 (PGK1)] change in ∆∆C_t. Thus the choice of reference gene matters. Use of a relatively unstable reference gene, such as PKG1, would prevent identification of a significant effect of P. aeruginosa on IL8 (Fig. 5).

Fig. 5.

IL8 response to Pseudomonas aeruginosa (PA14) using the reference gene on the y-axis (∆∆Ct). Gray bars indicate a significant response (P < 0.5, linear model). False indicates P ≥ 0.05. True indicates P < 0.05.

It is also important to note that, in addition to stability, the direction of change of a reference gene has a significant impact on the outcome of qPCR data analysis. For example, PGK1 C_t decreased with P. aeruginosa exposure (Fig. 6); thus, use of PGK1 as a reference gene underestimates the increase in IL8 gene expression compared with an ideal reference gene that is unchanged by P. aeruginosa. By comparison, ELF1 C_t increased with P. aeruginosa; thus, use of ELF1 as a reference gene overestimates the increase in IL8 gene expression compared with use of an ideal reference gene that is unchanged by P. aeruginosa (Fig. 6). The choice of a reference gene that is neither induced nor repressed by P. aeruginosa, e.g., RPL30 (Fig. 6), would provide a less biased estimate of IL8. In addition, use of the geometric mean of three reference most-stable genes selected by the NormqPCR R package (see methods) would produce a result similar to that with RPL30 alone (Fig. 6). A side-by-side plot, such as Fig. 6, demonstrates that RPL30 is the best of the three reference genes mentioned above for measurement of IL8.

Fig. 6.

Cycle threshold (Ct) values for IL8 and three candidate reference genes and the geometric mean of three stable reference genes (GeoMean) in cells from three donors exposed to Pseudomonas aeruginosa (PA14) or control for 1 h. PA14 was removed, and RNA was measured 24 h postinfection.

Common reference genes are more likely than other genes to be differentially expressed.

Our findings in airway epithelial cells raise the following question: Do reference genes respond to treatment conditions in general? To explore this possibility, we downloaded and analyzed gene expression data from 986 microarray studies that measured the expression of 6,356 well-annotated genes in human cells or tissues. Figure 7 shows the number of studies in which each of the 6,356 genes was differentially expressed. A typical reference gene was differentially expressed in just over 400 studies, as shown by the median line for the reference gene group in Fig. 7. Surprisingly, genes not included in Table 1 (“other genes”) tended to be differentially expressed less frequently than “reference genes,” although the two distributions overlap, as shown by the density plots in Fig. 7. However, large treatment responses were much less common when the best-of-10 reference genes in each study was identified. (“best reference” in Fig. 8).

Fig. 7.

Number of studies in which selected sets of genes achieved statistically significant treatment effects (P < 0.05). Each dot represents a gene. “Other gene” refers to one of 6,356 Kyoto Encyclopedia of Genes and Genomes genes, “reference gene” refers to genes listed in Table 1, and “best reference” refers to the least variable reference gene in each study. **Significant difference between other genes and reference genes (P = 0.004 by Wilcoxon’s rank sum test). Horizontal bars indicate median values. Width of the shaded area is proportional to the number of observations at a given height.

Fig. 8.

Maximum fold change between experimental conditions for genes and gene categories shown in Fig. 7. Each dot represents a gene. “Other gene” refers to one of 6,356 Kyoto Encyclopedia of Genes and Genomes genes, “reference gene” refers to genes listed in Table 1, and “best reference” refers to the least-variable reference gene in each study. ***Significant difference between other genes and reference genes (P < 2.2e-16 by Wilcoxon’s rank sum test). Horizontal bars indicate median value for genes in a gene category. Width of the shaded area is proportional to the number of observations at a given height.

Although reference genes achieve statistically significant responses more frequently than nonreference genes, the size of treatment responses was slightly less in reference genes than other genes. For example, the median (50th percentile) reference gene response to treatment was 21.1% compared with 26.3% for nonreference genes (Table 5). This shift toward smaller treatment responses in reference genes than nonreference genes was statistically significant (Fig. 8). In our analysis, the best reference gene in any experiment, defined as the reference gene with the smallest fold-change response to treatment condition, responded much less than other genes in the same quartile (Table 5).

Table 5.

Responses of other (nonreference) genes, 10 common reference genes (Table 1), and best reference genes

	25%	50%	75%	95%
Other gene	9.9	26.3	71.1	339.7
Reference gene	8.8	21.1	49.2	211.6
Best reference	1.4	4.4	11.8	36.6

Responses are expressed as maximum percent difference in expression between any 2 groups in an experiment for 3 quartiles in variability (25%, 50%, and 75%) and at the 95th percentile in their response to treatment conditions. For example, 95% of group differences for reference genes were <211.6%.

No single reference gene performs well in all experimental systems.

As shown in Table 5, the best reference gene in any of the 986 studies analyzed performed much better than a randomly selected reference gene. If the same gene performed well in all systems, one could simply select that ideal gene and use it in all cases. Consistent with previous reports (15, 18, 37, 52, 53), our analysis of 10 common reference genes in 986 studies showed that no single gene stood out as the least responsive to treatments. For example, as shown in Fig. 9, ACTB, a common reference gene, had a median maximum log₂ fold change of 0.27, which is equivalent to a maximum fold difference between groups of ~21% (Table 5). However, a visual inspection of Fig. 9 shows that all 10 common reference genes sometimes respond to treatment by ≥1 on a log₂ scale, i.e., a twofold difference between groups. Our analysis of 986 gene expression studies supports recommendations by others (4, 5, 13, 14, 17, 23, 24, 29, 37, 38, 43, 50) that the stability of qPCR reference genes should be experimentally demonstrated rather than assumed.

Fig. 9.

Response of 10 common reference genes in 986 independent studies. Each dot represents the response of a reference gene. Height on the y-axis represents the maximum fold change for a given reference gene (x-axis) between any two conditions measured in a study. Horizontal bars represent the median value for all 986 measurements of each gene, and width of the shaded area is proportional to the number of observations at a given height.

DISCUSSION

To the best of our knowledge, this is the most comprehensive gene expression analysis of reference genes in airway epithelial cells that are commonly used in qPCR studies. Our analysis of 15 airway epithelia microarray data sets, 986 human microarray data sets downloaded from the NCBI, and TaqMan analysis of 32 reference genes in primary cultures of HBEC exposed to P. aeruginosa reveals that reference gene instability is a general phenomenon but that relatively stable reference genes can be identified on a case-by-case basis. Our analysis highlights the importance of identifying adequate reference genes for each qPCR experiment. In addition, documentation of reference gene responses, together with responses in genes of interest in all publications, will enhance experimental rigor and reproducibility.

Stability of reference genes should not be assumed.

The terms “reference gene” and “housekeeping gene” are often used interchangeably. Although housekeeping genes are constitutively expressed and are vital for basic cellular function, there is no guarantee that they are expressed at similar levels, regardless of conditions, and, therefore, can serve as a reference gene for qPCR (15, 18). However, if a housekeeping gene is stably expressed, it can serve as reference gene for qPCR.

Several reports (5, 15, 17) have concluded that the lack of compliance with recommended Minimum Information for Publication of Quantitative Real-Time PCR Experiments (MIQE) guidelines for qPCR analysis (6) and lack of reference gene validation are likely to lead to significant errors. This conclusion is based largely on direct measurements of a small number of reference genes in different tissues (52) and in response to infection (16). Our meta-analysis of microarray data provides a much larger unbiased sample of how reference genes (Table 1) generally behave. We were surprised to find that the average reference gene was differentially expressed in 418 of 986 studies, that is, ~42% of the time (Fig. 7). This high rate of differential expression argues strongly against the assumption that a common reference gene will be stably expressed in a new experimental model.

Accurate detection of small responses in genes of interest is difficult unless a stable reference gene is identified.

The ability to identify changes in target genes accurately depends on the relative variability in the gene of interest and the reference gene selected. For example, 10 common reference genes responded by <21% in 50% of cases, meaning that although they may not be ideal, they would be sufficiently stable to serve as references where responses of genes of interest are much larger than 21% (Table 5). However, accurate measurement of small changes in genes of interest would require a reference gene with a relatively smaller response to treatment. For example, in 50% of experiments, the least responsive reference gene responded to treatment by <5%. Therefore, independent experiments to identify sufficiently stable reference genes are required to accurately identify small responses in genes of interest.

Costs and benefits of using reference gene panels.

Every decision in research is a trade-off between costs and benefits. In many cases, the cost of performing a TaqMan Array Human Endogenous Control panel to identify the best of 32 reference genes may be difficult to justify. For example, we spent about $5,000, including supplies and labor, to perform tests of reference gene stability for three cell donors in two experimental conditions. We calculated geometric mean of three stable reference genes (52) on this panel to create the geometric mean signal shown in Figs. 5 and 6. The geometric mean of these three genes would provide results similar to RPL30 alone. Although the TaqMan Array Human Endogenous Control panel would be useful to identify a stable reference gene, the cost may not justify its use in all experiments. Because multiple reference genes are rarely analyzed (9), we and others (8, 13, 22, 34, 38, 40), including MIQE guidelines (6), suggest that, for every qPCR experiment, studies should be conducted to evaluate several reference genes and as many as possible.

Implications for scientific rigor and reproducibility.

A major initiative of the National Institutes of Health (NIH) is to improve rigor and reproducibility in scientific research (42), and inadequate qPCR methodology has been cited as an example of a practice that may lead to unreproducible results (5). Some of the results in this study lend credence to these criticisms: reference genes are frequently differentially expressed, and it is not possible to predict the circumstances in which a given reference gene is stable. No author, reader, or reviewer should assume that differential gene expression measured in terms of a reference gene is valid without a description of how the reference gene was selected and how it performed in a given experiment. The requirement for explicit evaluation of reference genes in qPCR is not unlike the expectation in Western blot experiments that an internal control, such as actin, is presented beside the protein of interest to allow reviewers and readers to determine if the reference protein is affected by the experimental maneuver (46). On the other hand, one cannot assume, just because a published report failed to justify the choice of a reference gene or failed to report reference gene expression, that the results are invalid. Adherence to the extensive list of MIQE guidelines (6) in every research report would increase rigor and reproducibility, as would simple reporting of the reference gene responses, as in Fig. 6.

Conclusions.

In our analysis of ∼1,000 gene expression studies, we found that reference gene instability is a general phenomenon and not limited to studies of airway epithelial cells. These results highlight the importance of identifying adequate reference genes for each experimental system and documenting their response to treatment in each experiment. In an ideal world, reference gene selection would be based on resource-intensive approaches, such as reference gene panels. Use of a reference gene panel allows researchers to identify an optimal set of reference genes and facilitates more accurate quantification. High accuracy is of paramount importance when the aim is to quantify small differences in gene expression. However, regardless of the process used to select reference genes, explicit reporting of their performance in a given experiment enables readers and reviewers to judge for themselves whether a given reference was sufficiently stable to justify experimental conclusions. This will enhance experimental rigor and reproducibility, a major initiative of the NIH.

GRANTS

This work was supported by National Institute of Diabetes and Digestive and Kidney Diseases Grant P30-DK117469 and Cystic Fibrosis Foundation Grants STANTO19R0, STANTO19GO, and STANTO02PO.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

T.H.H., K.K., and B.A.S. conceived and designed research; T.H.H., K.K., and L.B. performed experiments; T.H.H. and K.K. analyzed data; T.H.H., K.K., and B.A.S. interpreted results of experiments; T.H.H. and K.K. prepared figures; T.H.H., K.K., and B.A.S. drafted manuscript; T.H.H., K.K., and B.A.S. edited and revised manuscript; T.H.H., K.K., and B.A.S. approved final version of manuscript.