Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Mar 24.
Published in final edited form as: Biotechnol Bioeng. 2009 Oct 1;104(2):390–399. doi: 10.1002/bit.22411

In Vitro and In Vivo RNA Interference Mediated Suppression of Tn-Caspase-1 for Improved Recombinant Protein Production in High Five Cell Culture With the Baculovirus Expression Vector System

Colin G Hebert 1,2,3, James J Valdes 2, William E Bentley 1,3,4
PMCID: PMC10960971  NIHMSID: NIHMS145718  PMID: 19557836

Abstract

While traditional metabolic engineering generally relies on the augmentation of specific genes and pathways in order to increase the yield of target proteins, the advent of RNA interference (RNAi) as a biological tool has given metabolic engineers another tool capable of rationally altering the host cell’s biological landscape in order to achieve a specific goal. Given its broad applicability and potent specificity, RNAi has the ability to suppress genes whose function is contrary to the desired phenotype. In this study, RNAi has been used to increase recombinant protein production in a Trichoplusia ni derived cell line (BTI-TN-5B1-4—High Five) using the Baculovirus Expression Vector System. The specific target investigated is Tn-caspase-1, a protease involved in apoptosis that is likely the principal effector caspase present in T. ni cells. Experiments were first conducted using in vitro synthesized dsRNA to verify silencing of Tn-capase-1 and increased protein production as a result. Subsequent experiments were conducted using a cell line stably expressing in vivo RNAi in the form of an inverted repeat that results in a hairpin upon transcription. Using this construct, Tn-caspase-1 transcript levels were decreased by 50% and caspase enzymatic activity was decreased by 90%. This cell line, designated dsTncasp-2, demonstrates superior viability under low nutrient culture conditions and resulted in as much as two times the protein yield when compared to standard High Five cells.

Keywords: RNA interference, RNAi, High Five, metabolic engineering, Baculovirus Expression Vector System, BEVS

Introduction

In general, metabolic engineering involves altering a gene or regulatory network to bring about a desired phenotype (Bailey, 1991; Stephanopoulos and Vallino, 1991). Traditionally, this has been accomplished either through the overexpression of specific targets directly involved in the pathway (Chen et al., 2001; Irani et al., 1999) or alteration of the overall protein synthesis machinery, such as the expression of chaperone proteins (Borth et al., 2005; Davis et al., 2000; Higgins et al., 2003; Kitchin and Flickinger, 1995; Yokoyama et al., 2000) or transcriptional factors (Tigges and Fussenegger, 2006) involved in the secretory pathway. However, as illustrated in Figure 1A, a similar strategy can be implemented using RNA interference (RNAi) (Hebert et al., 2008). First described in 1998 (Fire et al., 1998), RNAi involves a broad class of RNA molecules that can be used to suppress or “silence” homologous RNA through a pathway that is thought to be conserved throughout eukaryotes (Dykxhoorn and Lieberman, 2005; Fagard et al., 2000). The fundamental goal of metabolic engineering with RNAi remains the same, though the method of execution is different. Instead of introducing new genes or pathways, potentially deleterious genes and pathways are suppressed in an effort to optimize the host’s native production machinery.

Figure 1.

Figure 1

RNAi in metabolic engineering and experimental design. A: The goal of RNAi as a metabolic engineering tool is to suppress genes that would divert flux away from the product of interest. Potential target genes include enzymes involved in alternative pathways or that degrade the product of interest. B: Experimental design—in vitro dsRNA and baculovirus infection. dsRNA is first synthesized in vitro from a dsDNA template and subsequently transfected into High Five cells at a concentration of 1 μg/mL. The following day, cells are infected with a recombinant baculovirus. Throughout infection, RNA and proteins levels are monitored using RT-PCR and fluorescence measurements. C: Experimental design—in vivo dsRNA and baculovirus infection. Since the dsRNA construct is stably integrated, cells can be seeded and infected without a separate transfection step.

With respect to recombinant protein production, specific desirable phenotypes include the ability to produce a specific product at a high yield and quality, as well as the ability to maintain a high cell viability in the harsh environments generally seen during the production process, including decreased oxygen and nutrient concentrations or increased cellular and oxidative stress (diStefano et al., 1995; Rao and Bredesen, 2004; Scott et al., 1992). Efforts have been made using RNAi to increase protein production in Drosophila S2 cells through manipulation of the cell growth cycle (March and Bentley, 2006, 2007) and in HEK cells through the downregulation of transcriptional regulators (Hacker et al., 2004). Protein quality has also been improved by using RNAi to suppress specific genes, resulting in both increased antibody-dependent cellular cytotoxicity for anticancer therapeutics (Imai-Nishiya et al., 2007; Kanda et al., 2007; Mori et al., 2004), as well as higher sialic acid content in recombinant glycoproteins (Ngantung et al., 2006). Specific examples of antiapoptosis engineering involving RNAi include enhancing cell viability and protein production in CHO cells through the downregulation of either caspase-3 expression (Lai et al., 2004; Sung et al., 2005) or the expression of both caspase-3 and caspase-7 (Sung et al., 2007) as well as the suppression of the antiapoptotic genes Alg-2 and Requiem (Wong et al., 2006b). The latter study was the second article in a series that used transcriptional profiling to determine potential RNAi targets (Wong et al., 2006a). A similar screen is underway in HEK cells (Lee et al., 2007).

RNAi for increasing protein production in the Baculovirus Expression Vector System (BEVS) has been less common, despite its high productivity (Caron et al., 1990; Kim et al., 2007; Luckow and Summers, 1988; Smith et al., 1985). Initial reports in Trichoplusi ni larvae and Spodoptera frugiperda derived cell lines demonstrated that genes produced on a baculovirus could be silenced with RNAi, and subsequent applications of this technique aimed to reduce baculovirus infectivity through the downregulation of several key genes (Hajos et al., 1999; Isobe et al., 2004; Valdes et al., 2003). Relatively few studies have examined the potential of RNAi to increase protein production in the BEVS, though several examples have appeared in both cells and larvae demonstrating the effectiveness of the approach (Kim et al., 2007; Kramer and Bentley, 2003; Lin et al., 2007).

In a previous work (Kim et al., 2007), we demonstrated that protein production in the BEVS could be enhanced using in vitro dsRNA against baculovirus genes in Sf9 cells. Here, we demonstrate that the same principle holds true for High Five cells, which have been shown to have superior productivity (Davis et al., 1993; Wickham and Nemerow, 1993; Wickham et al., 1992). Furthermore, while previous studies have relied on in vitro synthesized dsRNA, this study demonstrates the first use of in vivo dsRNA produced by an engineered T. ni cell line to suppress a specific gene in order to improve recombinant protein production in the BEVS. This strategy is more appropriate for commercial applications that ideally involve a minimum number of processing steps (e.g., minimizing those that could introduce contamination such as adding dsRNA).

The metabolic engineering target was Tn-caspase-1, which has been shown to be the primary effector caspase produced by High Five cells and an accurate indication of the overall level of apoptosis present in the cells (Hebert et al., 2009). Although antiapoptotic proteins, such as Autographa californica multiple nucleopolyhedrovirus (AcMNPV) protein p35 (Clem et al., 1991), are produced by the baculovirus upon infection, caspase activity still increases nearly 10-fold at 5 days postinfection and beyond (Hebert et al., 2009), suggesting that the RNAi mediated suppression of Tn-caspase-1 (which prevents this increase in caspase activity during the late stages of baculovirus infection) could be an effective means of improving recombinant protein production in the BEVS.

In this study, we first show that protein production can be enhanced by silencing Tn-caspase-1 with chemically synthesized in vitro dsRNA (Fig. 1B). Further, in order to take advantage of the simplicity and robustness of an in vivo approach, we have developed a construct for the expression of in vivo dsRNA and subsequently developed a stable T. ni derived cell line in which it is implemented (Fig. 1C). The stable cell line selected (designated dsTncasp-2) displays a 50% decrease in caspase transcript level and a 90% decrease in caspase enzymatic activity during chemically induced apoptosis. dsTncasp-2 also has increased resistance to nutrient starvation and high cell densities as is often seen during prolonged suspension culture and demonstrates protein production levels up to twofold that of standard cells. In order to illustrate the generality of this approach, recombinant baculoviruses producing either of the two model proteins green fluorescent protein (GFP) and chloramphenicol acetyltransferase (CAT), were used to compare the production levels of dsTncasp-2 and control cells at several multiplicities of infection (MOI).

Materials and Methods

Cell Culture, Transfection, and Infection

T. ni BTI-TN-5B1-4 (High Five, Invitrogen, Carlsbad, CA) cells were cultivated in EX-CELL-405 medium (SAFC, St. Louis, MO) at 27°C. Unless otherwise noted, cells were initially plated on 24-well plates at 1 × 105 cells/well (2 × 105 cells/mL) and transfected the following day using FuGene HD® (Roche Applied Science, Indianapolis, IN) according to the manufacturer’s procedure. Briefly, DNA/FuGene complexes at a ratio of 2:7 (μg/μL) were formed in distilled and deionized water and then added to the appropriate wells. For studies involving in vitro synthesized dsRNA, 0.5 μg was added to each well (concentration of 1 μg/mL). In the case of infection with baculovirus, cells were infected 24 h posttransfection as previously described with a MOI of 5, unless otherwise noted (Kim et al., 2007). Cell counts and viability were determined using a hemocytometer and trypan blue exclusion.

Plasmid Construction and Selection of Stable Cell Line

The hairpin construct in pIB-dsTnCasp was created in two steps. Initially, the sense strand plus a 100 bp loop were PCR amplified from pIB-TnCasp (Hebert et al., 2009) using the following primers (5′-CGA TCA AAA TGC TGG ACG GTG-3′; 5′-GAA CCA CGA GTT GTG TTC CTC CAA-3′). This PCR fragment was then TOPO® cloned into pIB/V5-His-TOPO® (Invitrogen), which allows for the insertion of gene products under the control of the constitutive Opie-2 promoter and allows for the selection of stable cells using blasticidin, creating pIB-TnCasp-Sense. In a separate cloning step, the antisense fragment was first cloned into pCR®-Blunt II-Topo® (Invitrogen) using the following primers (5′-GCA CCG GTC GAT CAA AAT GCT GGA CGG TG-3′; 5′-GCC TCG AGA AGT CTG CAT GCA CAG GAA T-3′), creating pCR-TnCasp-Antisense. Next, pIB-dsTncasp-Sense, and pCR-dsTnCasp-Antisense were digested using the restriction enzymes AgeI and XhoI, and then ligated using T4 DNA Ligase (NEB) to create pIB-dsTnCasp which retains the blasticidin resistance gene and contains an insert that will form a RNA hairpin loop upon transcription. To create a stable cell line containing this construct, High Five cells were seeded in a 6-well plate at a density of 5 × 105 cells/mL and transfected with 2 μg of pIB-dsTnCasp. Two days post-transfection, cells were split at a ratio of 1:5 and subject to blasticidin selection at a concentration of 100 μg/mL based on the manufacturer’s recommendation. Five populations (two of which are discussed in the text) that showed resistance to blasticidin were selected and analyzed for Tn-caspase-1 transcript and protein levels.

Baculovirus

A recombinant AcMNPV that expresses GFPuv (a variant of wild-type Aequorea victoria GFP optimized for excitation by UV light (360–400 nm)—Clontech, Mountain View, CA) or CAT under the control of the polyhedron promoter was created previously (Cha et al., 1999a,b,c). The baculovirus was propagated S. frugiperda (Sf-9) cells (Invitrogen) (except for one amplification step that took place in High Five cells for the PPH-CAT virus) following the general protocols outlined in O’Reilly et al. (1992). All titers of baculovirus were determined in High Five cells by the endpoint dilution method.

In Vitro dsRNA Synthesis

Total RNA was extracted using the RNAqueous-4PCR Kit (Applied Biosystems/Ambion, Austin, TX) according to the manufacturer’s instructions. Cells were dislodged from the well plate by streaming media over them, spun at 500 g for 5 min, and the resulting cell pellet was lysed using the provided lysis/binding buffer. An equal volume of 64% ethanol was added to the mix before binding to a silica-based filter cartridge via centrifugation. The bound RNA was then washed three times using the provided wash solutions: once with wash solution 1 and twice with wash solution 2/3. The RNA was then eluted using two elution steps, using 50 and then 20 μL of the included Elution Solution. Following elution, samples were subjected to a DNAse digest (30 min at 37°C) using the included DNAse as per the manufacturer’s instructions in order to remove any contaminating DNA.

First strand templates of each target gene were then synthesized from 500 ng of total mRNA using oligo-dT primers and SuperScript III First-Strand Synthesis System for RT-PCR (Invitrogen). Regions (700–900 bp) of first strand DNA templates were PCR amplified by AccuPrime Taq DNA Polymerase High Fidelity (Invitrogen) using gene-specific primers for Tn-caspase-1 (5′-CGA TCA AAA TG[0]C TGG ACG GTG[0]-3′; 5′-AAG TCT GCA TGC ACA GGA AT 3′) or CAT (5′-ACT GGA TAT ACC ACC GTT GAT-3′; 5′-GTG CTG GAT ATC TGC AGA ATT-3′). T7 sequences (TAA TAC GAC TCA CTA TAG GGA) were added in a subsequent PCR step, resulting in T7 templates.

To synthesize the dsRNA, sense and antisense RNA was transcribed from T7 templates using the Megascript T7 kit (Ambion/Applied Biosystems) following the manufacturer’s instructions. Briefly, single-stranded RNA (ssRNA) was synthesized by mixing the T7 PCR template with the dNTPs and T7 RNA polymerase included in the kit and incubating for 4 h at 37°C. The ssRNA synthesized was extracted using phenol/chloroform, resuspended in nuclease free water, and then incubated at 65°C for 10 min before cooling to room temperature, annealing the two strands to form dsRNA. dsRNA was checked for size and integrity using agarose gel electrophoresis and then diluted to a concentration of 3 μg/μL.

Semi Quantitative RT-PCR

For determining relative transcript levels, RNA was extracted from High Five cells using an RNAqueous kit (Ambion) as described above. RNA concentration was determined by measuring the absorbance of a diluted sample at the 260 nm wavelength in a UV spectrometer (Beckman Coulter, Fullerton, CA). Five hundred nanograms of total RNA was subject to reverse transcription using oligo-dT primers to obtain the first-strand cDNA template. The cDNA template was subject to PCR with sequence-specific primers: Tn-caspase-1 (5′-TTC ATT CGA TCC CTG GAT AGC-3′; 5′ TAG TAT CCA GGC ACG GTG GAG-3′). PCR products were run on a 1% agarose gel stained with ethidium bromide to compare band intensities under fluorescent light. Primers against a 300 bp section of actin (5′-GAT ATG GAG AAG ATC TGG CA-3′; 5′-GCG TAG CCC TCG TAG ATG-3′) were used to PCR amplify a loading control from the reverse transcript. Photographs of agarose gels under UV light were taken with an AlphaImager® HP and quantitatively analyzed for band intensity using the “Spot Denso” function of the AlphaEaseFC software package (Alpha Innotech, San Leandro, CA). Exposure times were metered to ensure brightness values were not saturated.

Assay of Caspase Activity

After various experimental treatments, cells were collected and washed with PBS (pH 7.5). The cells were then suspended in a lysis buffer (from caspase 3 assay kit, BD Bioscience, San Jose, CA) and incubated on ice for 30 min. Cell debris was removed by centrifugation (10,000g for 5 min) and supernatants were stored for further use or assayed directly. The tetrapeptide N-acetyl-Asp.Glu.Va-l.Asp-7-amino-4-methylcoumarin (Ac-DEVD-AMC) was used as the substrate. Cell lysates were added to the reaction buffer (from caspase 3 assay kit, BD Bioscience) with Ac-DEVD-AMC and incubated for 4 h at 37°C. The release of AMC was measured using a Spectramax M5 Plate Reader (Molecular Devices, Sunnyvale, CA) (excitation λ380nm, emission λ440nm). Caspase activity is defined as the slope of the plot of fluorescence versus incubation time.

Analysis of GFP Expression

GFP fluorescence was measured using a SpectraMax M5 plate reader using an excitation wavelength of 395 nm and an emission wavelength of 509 nm. Fluorescence was measured in vivo using the well scan method of the SoftMax Pro software, which determines well fluorescence by averaging the fluorescence of 21 distinct points throughout the well. Thus, each well could be measured non-invasively and could be used for subsequent time points.

Analysis of CAT Activity

For each time point, cells were harvested and spun for 5 min at 500 g. The medium was then removed and saved for later analysis, and the cell pellet was resuspended in 100 μL Cytobuster (Novagen, Madison, WI) for 5 min. Cell lysates were then spun at 10,000g for 5 min to pellet the insoluble lysates. Samples were either analyzed directly or frozen at −20°C for later analysis. CAT was measured in both the extracellular medium and soluble cell lysates using a colorimetric assay. A reaction mix was prepared consisting of 100 μM Acetyl CoA (Sigma Aldrich, St. Louis, MO) and 1 mM 5,5′-dithiobis-(2-nitrobenzoic acid) in 0.1 M Tris–HCl buffer (pH 7.8). For each sample, 200 μL of the reaction mix was added to a well in a 96-well plate and allowed to reach 37°C. Then, an appropriate amount of sample (extracellular medium or lysates) was added to the well. Depending upon the concentration of CAT, samples were diluted 5- to 20-fold in order to fall within the linear range of the assay. Once the sample had been added and allowed to reach 37°C, 25 μL of 5 mM chloramphenicol was added to each well and the absorbance (412 nm) was measured over a period of several minutes. CAT activity was determined from the initial slope of the plot of OD versus time, and normalized by the amount of crude, undiluted sample. The reported CAT activity is a sum of the CAT present in the medium and in the cell lysate, scaled appropriately by the relative volume of each.

Results and Discussion

Silencing Tn-Caspase-1 Via In Vitro RNAi—Effects on Recombinant Protein Production

In a previous study (Hebert et al., 2009), we investigated the behavior of Tn-caspase-1 both under chemically induced apoptosis and during baculovirus infection. In addition to showing that Tn-caspase-1 is presumably the main effector caspase present in High Five cells, we demonstrated that the enzymatic activity of Tn-caspase-1 could be linked to and affect the overall level of apoptosis present in High Five cells, and that caspase activity was present during baculovirus infection, especially at 5 days postinfection and longer. Thus, an initial goal of this study was to see if in vitro suppression of Tn-caspase-1 increased recombinant protein production in the BEVS. An overall experimental scheme is shown in Figure 1B. dsRNA is first synthesized in vitro from dsDNA before it is transfected at a concentration of 1 μg/mL into High Five cells that have been seeded in 24-well plates the previous day at a density of 1 × 105 cells/well (2 × 105 cells/mL). In addition to dsRNA against the target gene, two additional conditions were used; no dsRNA and non-specific dsRNA (dsRNA synthesized against CAT). The day following transfection, cells were infected with a recombinant baculovirus. RNA and protein levels were measured during infection using RT-PCR (RNA) and fluorescence (GFP) measurements, respectively.

In order to determine the effectiveness of the dsRNA against Tn-caspase-1, total RNA was harvested from each group at 72 h postinfection and analyzed using RT-PCR and differential display. Results are presented in Figure 2. Infected cells and cells treated with non-specific dsRNA against CAT have ~10% lower Tn-caspase-1 transcript levels, but this difference is not statistically significant (P < 0.05). Cells transfected with dsRNA against Tn-caspase-1 show a much larger and significant decrease in transcript level, a nearly 80% decrease when compared to non-transfected cells, demonstrating that Tn-caspase-1 is successfully silenced at the transcriptional level.

Figure 2.

Figure 2

Suppression of Tn-caspase-1 via in vitro RNAi during baculovirus infection. Treatment groups included uninfected, infected (no dsRNA), infected + dsRNA against CAT, and infected + dsRNA against Tn-caspase-1. A: RT-PCR showing transcript levels of both Tn-caspase-1 and actin. Bands were visualized using ethidium bromide under exposure to UV light. B: Graphical representation of the relative transcript levels of Tn-caspase-1 under all conditions, normalized to the level of actin transcript at each condition. C: GFP production levels from days 2 to 6 postinfection during suppression of Tn-caspase-1 with in vitro dsRNA. An additional control consisting of only the transfection reagent (Fugene Only) is included. Error bars represent the standard deviation for at least three independent experiments. Asterisks (*) indicate a significant (P < 0.05) difference between cells treated with Tn-caspase-1 dsRNA and all control conditions (B and 2). Asterisk (**) indicates a significant difference between cells treated with Tn-caspase-1 dsRNA and infected cells (C only).

Once suppression at the transcript level was confirmed, a recombinant baculovirus producing GFP was used to determine if protein production levels were affected by the silencing. Controls of non-specific dsRNA (dsCAT) and no dsRNA (Infected) were included, along with an additional control of only transfection reagent (Fugene Only). As shown in Figure 2C, all transfection conditions demonstrated higher protein levels than non-transfected cells from days 3 to 6. However, the highest protein level at those time points was in cells transfected with dsRNA against Tn-caspase-1, which was significantly (P < 0.05) higher than infected cells from days 3 to 6 and just over 2.5 times higher than infected cells at days 5 and 6. Protein levels in Tn-caspase-1 suppressed cells were also significantly higher (days 4 and 6) than those groups of cells receiving only transfection reagent or non-specific dsRNA, indicating that the increase was indeed due to a sequence specific effect. This demonstrates that in vitro RNAi against Tn-caspase-1 is an effective means of increasing recombinant protein production in the baculovirus system.

Development of Tn-Caspase-1 Suppressed Cell Line

Once the beneficial effects of Tn-caspase-1 suppression were verified using in vitro dsRNA, steps were taken in order to create a cell line capable of expressing in vivo dsRNA against Tn-caspase-1. This approach has several advantages, including elimination of the non-specific effects due to the transfection process, minimal perturbation of the cells during the infection and production process, and a reduction in materials requirements since there is no need to chemically synthesize dsRNA. Using the pIB/V5-His-TOPO® plasmid as a backbone, a ~600 bp region of Tn-caspase-1 plus a 100 bp spacer was PCR amplified from pIB-TnCasp (Hebert et al., 2009) and cloned into the vector in sense orientation to create pIB-Tncasp-Sense. The same region was PCR amplified using a different set of primers to create pCR-dsTncasp-Antisense, which contains the 600 bp region of Tn-caspase-1 in the opposite orientation. The antisense fragment was then digested using the restriction enzymes AgeI and Xho1 and subsequently ligated into pIB-dsTncasp-Sense, forming pIB-dsTncasp (Fig. 3). Cells were then transfected with pIB-dsTnCasp and transformants were selected using the antibiotic blasticidin.

Figure 3.

Figure 3

Plasmid map and detail of the in vivo dsRNA plasmid for the suppression of Tn-caspase-1. A: Plasmid map of pIB-dsTncasp, which contains a 600 bp portion of Tn-caspase-1 in the sense and antisense orientation separated by a 100 bp spacer, all under the control of the constitutive Opie-2 promoter. pIB-dsTncasp also contains a blasticidin resistance gene under the control of the Opie-1 promoter. B: Detail of the dsRNA construct. Upon transcription, the mRNA transcript folds upon itself, forming a dsRNA hairpin, which is recognized by dicer and subsequently processed into siRNA.

Five populations of blasticidin resistant cells (two of which were designated dsTncasp-1 and dsTncasp-2) were then tested for their level of Tn-caspase-1 transcription and enzymatic activity. Results for these two populations are shown in Figure 4. Both dsTncasp-1 and dsTncasp-2 show a reduction in Tn-caspase-1 transcription levels as shown via RT-PCR (Fig. 4A). However, dsTncasp-2 has a greater decrease, at 50% versus 40% in dsTncasp-1 (Fig. 4B). This difference is more apparent when examining the level of caspase enzymatic activity present in dsTncasp-1 and dsTncasp-2 versus control cells (Fig. 4C). As demonstrated previously, Tn-caspase-1 and other caspases have minimal activity unless cells are exposed to an apoptotic stimulus, such as the chemical apoptosis inducer actinomycin D (Ahmad et al., 1997; Hebert et al., 2009; LaCount et al., 2000). Thus, the caspase activity of both dsTncasp-1 and dsTncasp-2 was compared to control cells under normal conditions and after exposure to actinomycin D (24 h at 2 μg/mL). As shown in Figure 4C, the caspase activity levels of untreated cells are statistically similar for each cell line. However, when treated with actinomycin D, the caspase activity of control cells jumps to nearly 70 times control levels, while both dsTncasp-1 and dsTncasp-2 show a much smaller increase; 25 and 7 times control levels, respectively. In other words, dsTncasp-1 has a 60% decrease in caspase activity levels when treated with actinomycin D, while dsTncasp-2 shows a nearly 90% decrease when compared to control cells. So while the level of suppression at the transcript level is relatively modest when compared to in vitro dsRNA, suppression at the protein level, which is arguably more important, remains quite effective. This large increase in level of suppression when moving from transcript to protein could be explained by the self-activating nature of caspases. Because Tn-caspase-1 presumably plays the role of both affecting apoptosis as well as activating itself, a given reduction in the available amount of inactive Tn-caspase-1 would lead to a proportionally greater decrease in the amount of active caspase. Since dsTncasp-2 showed a greater level of suppression than dsTncasp-1, it was selected for subsequent experiments examining growth, viability, and protein production.

Figure 4.

Figure 4

In vivo suppression of Tn-caspase-1—RNA and protein levels. A: RT-PCR of Tn-caspase-1 and actin transcript levels in control High Five cells versus two stable cell populations, dsTnCasp-1 and dsTncasp-2. Bands were visualized using ethidium bromide under exposure to UV light. B: Graphical representation of the relative transcript levels of Tn-caspase-1 in standard High Five cells versus stable cell populations, normalized to the level of actin transcript at each condition. C: Caspase enzymatic activity of standard High Five cells versus the two stable cell populations. Caspase activity was measured from cell lysates in the presence or absence of actinomycin D (24 h at 2 μg/mL) and is normalized for each condition to untreated control cells. Error bars represent the standard deviation for at least three independent experiments. Asterisk (*) indicates a significant (P < 0.05) difference between the caspase activity of dsTncasp-1 and dsTncasp-2 cells versus control cells.

Growth and Viability of Stable Cell Line

Once the suppression of Tn-caspase-1 had been verified, experiments were conducted in order to test the growth characteristics of dsTncasp-2 against control cells. Both lines were seeded at a concentration of 5 × 105 cells/mL in 60 mm2 culture dishes (5 mL total volume). A total of four culture conditions were used. For both control and dsTncasp-2 cells, one group received 500 μL of medium each day before counting, while the other group received 500 μL of PBS (pH 7.4), in order to maintain a consistent culture volume. Cell count and viability was determined each day for 7 days postseeding. Results are presented in Figure 5.

Figure 5.

Figure 5

Cell growth and viability in suspension culture. Treatment groups included control, control + medium, dsTncasp, and dsTncasp + medium. In “+ medium” groups, 500 μL of fresh medium was added each day before counting. In other groups, 500 μL of PBS buffer was added each day before counting to maintain a consistent volume. A: Viable cell count of each treatment group, determined using a hemocytometer. B: Viability of each treatment group, determined by trypan blue exclusion. Error bars represent the standard deviation of the mean for at least three independent experiments.

From days 0 to 3, all four groups of cells show statistically similar growth rates (average doubling time between 27 and 29 h over the entire 3-day period), but differences between the conditions appear on day 4 and beyond. The viable cell count for control cells, both with and without supplemental medium, drops sharply from days 4 to 7. Cells that did not receive medium fare the worst, dropping from an average density of ~2.6 × 106 cells/mL at day 3 to ~1.4 × 106 cells/mL at day 4 and then finally to ~6 × 105 cells/mL at day 7, only just above the number of cells at seeding. Those cells that did receive medium fare better than those without, but still drop steadily from ~2.9 × 106 cells/mL at day 3 to ~1.3 × 106 cells/mL at day 7. dsTncasp-2 cells with and without additional medium show greater resilience to the adverse culture conditions. dsTncasp-2 cells that only received PBS did decrease in cell count from days 3 to 7, but still maintained a higher cell count than both groups of control cells. dsTncasp-2 cells that did receive additional medium have similar cell counts from days 4 to 6, only pulling ahead of the PBS cells at day 7. At day 7, dsTncasp-2 cells, both with and without additional medium, show a significantly (P < 0.05) higher cell count than both groups of standard cells.

Viability results show a similar trend. From days 1 to 3 postseeding, all four groups of cells maintain viability ranging between 96 and 97 ± 3% (95% confidence interval). However, from days 4 to 7, both groups of control cells show a rapid decline in viability, dropping to 24 ± 21% (PBS) and 58 ± 5% (medium) on day 7. In contrast, dsTncasp-2 cells, especially those that received additional medium, maintain higher viability, remaining at 65 ± 3% (PBS) and 82 ± 7% (medium) on day 7.

Since low nutrient concentrations can result in cell death (Goswami et al., 1999; Mastrangelo and Betenbaugh, 1998), it is possible that the decreased caspase activity present in the dsTncasp-2 cells helps prevent this, allowing the cells to maintain higher cell count and viability. The suppression of Tn-caspase-1 seems to be a more effective means of preventing cell death than the addition of medium. It is interesting to note that while the addition of medium clearly benefits both the cell count and viability of the control group of cells, the effect was less dramatic in the dsTncasp-2 cells, which only showed a significant enhancement of cell count and viability at day 7. It is possible that since dsTncasp-2 cells have a higher tolerance for poor culture conditions, it takes a longer time for the lack of medium to have an appreciable impact, thus delaying the onset of beneficial effects due to additional medium. Overall, these results demonstrate that the silencing of Tn-caspase-1 is an effective means of maintaining cell count and viability in adverse culture conditions.

Recombinant Protein Production With dsTncasp-2 Cell Line

In order to test whether the suppression of Tn-caspase-1 via in vivo dsRNA would lead to increased recombinant protein production, both control and dsTncasp-2 cells were infected with a recombinant baculovirus producing GFP, this time at three different MOIs: 0.1, 1, and 10. Expression levels were measured from days 2 to 6 and are presented in Figure 6A. From days 3 to 6, GFP levels at all MOIs are significantly (P < 0.05) higher for dsTncasp-2 cells versus control cells, averaging about 1.5 times greater at MOI 0.1 and 1 and 2 times greater at MOI 10. In order to further investigate the performance of dsTncasp-2 cells, additional experiments were conducted using a recombinant baculovirus that produces CAT. Both control and dsTncasp-2 cells were again infected at MOIs 0.1, 1, and 10. Results are shown in Figure 6B. Although the difference is not as dramatic as with GFP, dsTncasp-2 cells again show higher levels of recombinant protein, especially at 6 days postinfection, when both MOI 1 and 10 show a significant increase in CAT production (2.4× and 1.6×, respectively, P < 0.05).

Figure 6.

Figure 6

Protein production of dsTnCasp-2 versus control cells while varying the MOI of a recombinant baculovirus producing either GFP or CAT. A: GFP production at MOI 0.1, 1.0, and 10 from 2 to 6 days postinfection. B: CAT production at MOI 0.1, 1.0, and 10 from 3 to 6 days postinfection. GFP was measured in vivo using a fluorescence plate reader, while CAT activity was determined using the enzymatic assay described in Materials and Methods Section. Error bars represent standard deviations for each time point for at least three independent experiments. Asterisk (*) indicates a significant (P < 0.05) difference between the infected and dsTnCasp-2 cells at that time point and MOI.

Overall, our results demonstrate that suppression of Tn caspase via in vitro dsRNA is an effective means of increasing protein production in the BEVS. Further, we successfully created several cell lines expressing a dsRNA hairpin that resulted in the constitutive suppression of Tn-caspase-1 at both the transcript and protein levels. Based on its high level of suppression, dsTncasp-2 was chosen for studies examining growth, viability and protein production. dsTncasp-2 cells demonstrated an ability to maintain high cell densities and viability under low nutrient conditions, as well as superior protein productivity upon infection with recombinant baculoviruses producing both GFP and CAT. Finally, this work continues to support the application of RNAi in metabolic engineering for the improvement of recombinant protein production and the development of cell lines with desirable production phenotypes.

Acknowledgments

Contract grant sponsor: National Institutes of Health

Contract grant number: GM70851-01

Partial support of this work was provided by the United States Department of Defense (CREST Fellowship) and the National Institutes of Health (GM70851-01).

References

  1. Ahmad M, Srinivasula SM, Wang LJ, Litwack G, Fernandes-Alnemri T, Alnemri ES. Spodoptera frugiperda caspase-1, a novel insect death protease that cleaves the nuclear immunophilin FKBP46, is the target of the baculovirus antiapoptotic protein p35. J Biol Chem. 1997;272(3):1421–1424. doi: 10.1074/jbc.272.3.1421. [DOI] [PubMed] [Google Scholar]
  2. Bailey JE. Toward a science of metabolic engineering. Science. 1991;252:1668–1675. doi: 10.1126/science.2047876. [DOI] [PubMed] [Google Scholar]
  3. Borth N, Mattanovich D, Kunert R, Katinger H. Effect of increased expression of protein disulfide isomerase and heavy chain binding protein on antibody secretion in a recombinant CHO cell line. Biotechnol Prog. 2005;21(1):106–111. doi: 10.1021/bp0498241. [DOI] [PubMed] [Google Scholar]
  4. Caron AW, Archambault J, Massie B. High-level recombinant protein production in bioreactors using the baculovirus-insect cell expression system. Biotechnol Bioeng. 1990;36:1133–1141. doi: 10.1002/bit.260361108. [DOI] [PubMed] [Google Scholar]
  5. Cha HJ, Dalal NG, Pham MQ, Bentley WE. Purification of human interleukin-2 fusion protein produced in insect larvae is facilitated by fusion with green fluorescent protein and metal affinity ligand. Biotechnol Prog. 1999a;15(2):283–286. doi: 10.1021/bp990022t. [DOI] [PubMed] [Google Scholar]
  6. Cha HJ, Dalal NG, Pham MQ, Vakharia VN, Rao G, Bentley WE. Insect larval expression process is optimized by generating fusions with green fluorescent protein. Biotechnol Bioeng. 1999b;65(3):316–324. doi: 10.1002/(sici)1097-0290(19991105)65:3<316::aid-bit9>3.0.co;2-x. [DOI] [PubMed] [Google Scholar]
  7. Cha HJ, Dalal NG, Vakharia VN, Bentley WE. Expression and purification of human interleukin-2 simplified as a fusion with green fluorescent protein in suspended Sf-9 insect cells. J Biotechnol. 1999c;69(1):9–17. doi: 10.1016/s0168-1656(98)00190-4. [DOI] [PubMed] [Google Scholar]
  8. Chen KQ, Liu Q, Xie LZ, Sharp PA, Wang DIC. Engineering of a mammalian cell line for reduction of lactate formation and high monoclonal antibody production. Biotechnol Bioeng. 2001;72(1):55–61. doi: 10.1002/1097-0290(20010105)72:1<55::aid-bit8>3.0.co;2-4. [DOI] [PubMed] [Google Scholar]
  9. Clem RJ, Fechheimer M, Miller LK. Prevention of apoptosis by a baculovirus gene during infection of insect cells. Science. 1991;254(5036):1388–1390. doi: 10.1126/science.1962198. [DOI] [PubMed] [Google Scholar]
  10. Davis TR, Wickham TJ, McKenna KA, Granados RR, Shuler ML, Wood HA. Comparative recombinant protein production of eight insect cell lines. In Vitro Cell Dev Biol. 1993;29A:388–390. doi: 10.1007/BF02633986. [DOI] [PubMed] [Google Scholar]
  11. Davis R, Schooley K, Rasmussen B, Thomas J, Reddy P. Effect of PDI overexpression on recombinant protein secretion in CHO cells. Biotechnol Prog. 2000;16(5):736–743. doi: 10.1021/bp000107q. [DOI] [PubMed] [Google Scholar]
  12. diStefano DJ, Mark GE, Robinson DK. Feedings of nutrients delays apoptotic death in fed batch cultures of NS0 myeloma cells. Biotechnol Lett. 1995;18:1067–1072. [Google Scholar]
  13. Dykxhoorn DM, Lieberman J. The silent revolution: RNA interference as a basic biology, research tool, and therapeutic. Annu Rev Med. 2005;56:401–423. doi: 10.1146/annurev.med.56.082103.104606. [DOI] [PubMed] [Google Scholar]
  14. Fagard M, Boutet S, Morel JB, Bellini C, Vaucheret H. AGO1, QDE-2, and RDE-1 are related proteins required for post-transcriptional gene silencing in plants, quelling in fungi, and RNA interference in animals. Proc Natl Acad Sci. 2000;97:11650–11654. doi: 10.1073/pnas.200217597. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Fire A, Xu S, Montgomery MK, Kostas SA, Driver SE, Mello CC. Potent and specific genetic interference by double-stranded RNA in Caenorhabditis elegans. Nature. 1998;391:806–811. doi: 10.1038/35888. [DOI] [PubMed] [Google Scholar]
  16. Goswami J, Sinskey AJ, Steller H, Stephanopoulos GN, Wang DI. Apoptosis in batch cultures of Chinese hamster ovary cells. Biotechnol Bioeng. 1999;62(6):632–640. doi: 10.1002/(sici)1097-0290(19990320)62:6<632::aid-bit2>3.0.co;2-i. [DOI] [PubMed] [Google Scholar]
  17. Hacker DL, Bertschinger M, Baldi L, Wurm FM. Reduction of adenovirus E1A mRNA by RNAi results in enhanced recombinant protein expression in transiently transfected HEK293 cells. Gene. 2004;341:227–234. doi: 10.1016/j.gene.2004.06.054. [DOI] [PubMed] [Google Scholar]
  18. Hajos JP, Vermunt AMW, Zuidema D, Kulcsar P, Varjas L, de Kort CAD, Zavodszky P, Vlak JM. Dissecting insect development: Baculovirus-mediated gene silencing in insects. Insect Mol Biol. 1999;8(4):539–544. doi: 10.1046/j.1365-2583.1999.00150.x. [DOI] [PubMed] [Google Scholar]
  19. Hebert CG, Valdes JJ, Bentley WE. Beyond silencing—Engineering applications of RNA interference and antisense technology for altering cellular phenotype. Curr Opin Biotechnol. 2008;19(5):500–505. doi: 10.1016/j.copbio.2008.08.006. [DOI] [PubMed] [Google Scholar]
  20. Hebert CG, Valdes JJ, Bentley WE. Investigating apoptosis: Characterization and analysis of Trichoplusia ni caspase-1 through over-expression and RNAi mediated silencing. Insect Biochem Mol Biol. 2009;39(5):113–124. doi: 10.1016/j.ibmb.2008.10.009. [DOI] [PubMed] [Google Scholar]
  21. Higgins MK, Demir M, Tate CG. Calnexin co-expression and the use of weaker promoters increase expression of correctly assembled shaker potassium channel in insect cells. Biochim Biophys Acta. 2003;1610:124–132. doi: 10.1016/s0005-2736(02)00715-0. [DOI] [PubMed] [Google Scholar]
  22. Imai-Nishiya H, Mori K, Inoue M, Wakitani M, Iida S, Shitara K, Satoh M. Double knockdown of alpha 1,6-fucosyltransferase (FUT8) and GDP-mannose 4,6-dehydratase (GMD) in antibody-producing cells: A new strategy for generating fully non-fucosylated therapeutic antibodies with enhanced ADCC. BMC Biotechnol. 2007;7(84):1–13. doi: 10.1186/1472-6750-7-84. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Irani N, Wirth M, van den Heuvel J, Wagner R. Improvement of the primary metabolism of cell cultures by introducing a new cytoplasmic pyruvate carboxylase reaction. Biotechnol Bioeng. 1999;66(4):238–246. doi: 10.1002/(sici)1097-0290(1999)66:4<238::aid-bit5>3.0.co;2-6. [DOI] [PubMed] [Google Scholar]
  24. Isobe R, Kojima K, Matsuyama T, Quan GX, Kanda T, Tamura T, Sahara K, Asano SI, Bando H. Use of RNAi technology to confer enhanced resistance to BmNPV on transgenic silkworms. Arch Virol. 2004;149(10):1931–1940. doi: 10.1007/s00705-004-0349-0. [DOI] [PubMed] [Google Scholar]
  25. Kanda Y, Imai-Nishiya H, Kuni-Kamochi R, Mori K, Inoue M, Kitajima-Miyama K, Okazaki A, Iida S, Shitara K, Satoh M. Establishment of a GDP-mannose 4,6-dehydratase (GMD) knockout host cell line: A new strategy for generating completely non-fucosylated recombinant therapeutics. J Biotechnol. 2007;130(3):300–310. doi: 10.1016/j.jbiotec.2007.04.025. [DOI] [PubMed] [Google Scholar]
  26. Kim EJ, Kramer SF, Hebert CG, Valdes JJ, Bentley WE. Metabolic engineering of the baculovirus expression system via inverse “shotgun” genomic analysis and RNA interference (dsRNA) increases product yield and cell longevity. Biotechnol Bioeng. 2007;98(3):645–654. doi: 10.1002/bit.21353. [DOI] [PubMed] [Google Scholar]
  27. Kitchin K, Flickinger MC. Alteration of hybridoma viability and antibody secretion in transfectomas with inducible overexpression of protein disulfide-isomerase. Biotechnol Prog. 1995;11(5):565–574. doi: 10.1021/bp00035a011. [DOI] [PubMed] [Google Scholar]
  28. Kramer SF, Bentley WE. RNA interference as a metabolic engineering tool: Potential for in vivo control of protein expression in an insect larval model. Metab Eng. 2003;5(3):183–190. doi: 10.1016/s1096-7176(03)00027-2. [DOI] [PubMed] [Google Scholar]
  29. LaCount DJ, Hanson SF, Schneider CL, Friesen PD. Caspase inhibitor P35 and inhibitor of apoptosis Op-IAP block in vivo proteolytic activation of an effector caspase at different steps. J Biol Chem. 2000;275(21):15657–15664. doi: 10.1074/jbc.M000791200. [DOI] [PubMed] [Google Scholar]
  30. Lai DZ, Fu L, Weng SJ, Qi LQ, Yu CM, Yu T, Wang HT, Chen W. Blocking caspase-3 activity with a U6SnRNA promoter-driven ribozyme enhances survivability of CHO cells cultured in low serum medium and production of interferon-beta. Biotechnol Bioeng. 2004;85(1):20–28. doi: 10.1002/bit.10769. [DOI] [PubMed] [Google Scholar]
  31. Lee YY, Wong KTK, Nissom PM, Wong DCF, Yap MGS. Transcriptional profiling of batch and fed-batch protein-free 293-HEK cultures. Metab Eng. 2007;9(1):52–67. doi: 10.1016/j.ymben.2006.08.006. [DOI] [PubMed] [Google Scholar]
  32. Lin CC, Hsu JTA, Huang KL, Tang HK, Shu CW, Lai YK. Sf-Caspase-I-repressed stable cells: Resistance to apoptosis and augmentation of recombinant protein production. Biotechnol Appl Biochem. 2007;48:11–19. doi: 10.1042/BA20070044. [DOI] [PubMed] [Google Scholar]
  33. Luckow VA, Summers MD. Trends in the development of baculovirus expression vectors. Bio/Technology. 1988;6:47–55. [Google Scholar]
  34. March JC, Bentley WE. Engineering eukaryotic signal transduction with RNAi: Enhancing Drosophila S2 cell growth and recombinant protein synthesis via silencing of TSC1. Biotechnol Bioeng. 2006;95(4):645–652. doi: 10.1002/bit.20951. [DOI] [PubMed] [Google Scholar]
  35. March JC, Bentley WE. RNAi-based tuning of cell cycling in Drosophila S2 cells—Effects on recombinant protein yield. Appl Microbiol Biotechnol. 2007;73(5):1128–1135. doi: 10.1007/s00253-006-0560-x. [DOI] [PubMed] [Google Scholar]
  36. Mastrangelo AJ, Betenbaugh MJ. Overcoming apoptosis: New methods for improving protein-expression systems. Trends Biotechnol. 1998;16(2):88–95. doi: 10.1016/s0167-7799(97)01159-1. [DOI] [PubMed] [Google Scholar]
  37. Mori K, Kuni-Karnochi R, Yarnane-Ohnuki N, Wakitani M, Yamano K, Imai H, Kanda Y, Niwa R, Iida S, Uchida K, et al. Engineering Chinese hamster ovary cells to maximize effector function of produced antibodies using FUT8 siRNA. Biotechnol Bioeng. 2004;88(7):901–908. doi: 10.1002/bit.20326. [DOI] [PubMed] [Google Scholar]
  38. Ngantung FA, Miller PG, Brushett FR, Tang GL, Wang DIC. RNA interference of sialidase improves glycoprotein sialic acid content consistency. Biotechnol Bioeng. 2006;95(1):106–119. doi: 10.1002/bit.20997. [DOI] [PubMed] [Google Scholar]
  39. O’Reilly D, Miller LK, Luckow VA. Baculovirus expression vectors: A laboratory manual. New York: Freeman; 1992. [Google Scholar]
  40. Rao RV, Bredesen DE. Misfolded proteins, endoplasmic reticulum stress and neurodegeneration. Curr Opin Cell Biol. 2004;16:653–663. doi: 10.1016/j.ceb.2004.09.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Scott RL, Blanchard JH, Ferguson CHR. Effects of oxygen on recombinant protein production by suspension cultures of Spodoptera frugiperda (Sf-9) insect cells. Enzyme Microb Technol. 1992;14(10):798–804. doi: 10.1016/0141-0229(92)90095-6. [DOI] [PubMed] [Google Scholar]
  42. Smith GE, Ju G, Ericson BL, Moschera J, Lahm HW, Chizzonite R, Summers MD. Modification and secretion of human interlukin 2 produced in insect cells by a baculovirus expression vector. Proc Natl Acad Sci. 1985;82:8404–8408. doi: 10.1073/pnas.82.24.8404. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Stephanopoulos G, Vallino JJ. Network rigidity and metabolic engineering in metabolite overproduction. Science. 1991;252:1675–1681. doi: 10.1126/science.1904627. [DOI] [PubMed] [Google Scholar]
  44. Sung YH, Hwang SJ, Lee GM. Influence of down-regulation of caspase-3 by siRNAs on sodium-butyrate-induced apoptotic cell death of Chinese hamster ovary cells producing thrombopoietin. Metab Eng. 2005;7(5–6):457–466. doi: 10.1016/j.ymben.2005.08.001. [DOI] [PubMed] [Google Scholar]
  45. Sung YH, Lee JS, Park SH, Koo J, Lee GM. Influence of co-down-regulation of caspase-3 and caspase-7 by siRNAs on sodium butyrate-induced apoptotic cell death of Chinese hamster ovary cells producing thrombopoietin. Metab Eng. 2007;9(5–6):452–464. doi: 10.1016/j.ymben.2007.08.001. [DOI] [PubMed] [Google Scholar]
  46. Tigges M, Fussenegger M. Xbp1-based engineering of secretory capacity enhances the productivity of Chinese hamster ovary cells. Metab Eng. 2006;8(3):264–272. doi: 10.1016/j.ymben.2006.01.006. [DOI] [PubMed] [Google Scholar]
  47. Valdes VJ, Sampieri A, Sepulveda J, Vaca L. Using double-stranded RNA to prevent in vitro and in vivo viral infections by recombinant baculovirus. J Biol Chem. 2003;278(21):19317–19324. doi: 10.1074/jbc.M212039200. [DOI] [PubMed] [Google Scholar]
  48. Wickham TJ, Nemerow GR. Optimization of growth methods and recombinant protein production in BTI-Tn-5B1-4 insect cells using the baculovirus expression system. Biotechnol Prog. 1993;9:25–30. doi: 10.1021/bp00019a004. [DOI] [PubMed] [Google Scholar]
  49. Wickham TJ, Davis T, Granados RR, Shuler ML, Wood HA. Screening of insect cell lines for the production of recombinant proteins and infectious virus in the baculovirus expression system. Biotechnol Prog. 1992;8:391–396. doi: 10.1021/bp00017a003. [DOI] [PubMed] [Google Scholar]
  50. Wong DCF, Wong KTK, Lee YY, Morin PN, Heng CK, Yap MGS. Transcriptional profiling of apoptotic pathways in batch and fed-batch CHO cell cultures. Biotechnol Bioeng. 2006a;94(2):373–382. doi: 10.1002/bit.20872. [DOI] [PubMed] [Google Scholar]
  51. Wong DCF, Wong KTK, Nissom PM, Heng CK, Yap MGS. Targeting early apoptotic genes in batch and fed-batch CHO cell cultures. Biotechnol Bioeng. 2006b;95(3):350–361. doi: 10.1002/bit.20871. [DOI] [PubMed] [Google Scholar]
  52. Yokoyama N, Hirata M, Ohtsuka K, Nishiyama Y, Fujii K, Fujita M, Kuzushima K, Kiyono T, Tsurumi T. Co-expression of human chaperone Hsp70 and Hsdj or Hsp40 co-factor increases solubility of overexpressed target proteins in insect cells. Biochim Biophys Acta Gene Struct Expr. 2000;1493(1–2):119–124. doi: 10.1016/s0167-4781(00)00170-6. [DOI] [PubMed] [Google Scholar]

RESOURCES