Abstract
The emergence of drug-resistant influenza A virus (IAV) strains represents a serious threat to global human health and underscores the need for novel approaches to anti-influenza chemotherapy. Combination therapy with drugs affecting different IAV targets represents an attractive option for influenza treatment. We have previously shown that the thiazolide anti-infective nitazoxanide (NTZ) inhibits H1N1 IAV replication by selectively blocking viral hemagglutinin maturation. Herein we investigate the anti-influenza activity of NTZ against a wide range of human and avian IAVs (H1N1, H3N2, H5N9, H7N1), including amantadine-resistant and oseltamivir-resistant strains, in vitro. We also investigate whether therapy with NTZ in combination with the neuraminidase inhibitors oseltamivir and zanamivir exerts synergistic, additive, or antagonistic antiviral effects against influenza viruses. NTZ was effective against all IAVs tested, with 50% inhibitory concentrations (IC50s) ranging from 0.9 to 3.2 μM, and selectivity indexes (SIs) ranging from >50 to >160, depending on the strain and the multiplicity of infection (MOI). Combination therapy studies were performed in cell culture-based assays using A/Puerto Rico/8/1934 (H1N1), A/WSN/1933 (H1N1), or avian A/chicken/Italy/9097/1997 (H5N9) IAVs; dose-effect analysis and synergism/antagonism quantification were performed using isobologram analysis according to the Chou-Talalay method. Combination index (CI) analysis indicated that NTZ and oseltamivir combination treatment was synergistic against A/Puerto Rico/8/1934 (H1N1) and A/WSN/1933 (H1N1) IAVs, with CI values ranging between 0.39 and 0.63, independently of the MOI used. Similar results were obtained when NTZ was administered in combination with zanamivir (CI = 0.3 to 0.48). NTZ-oseltamivir combination treatment was synergistic also against the avian A/chicken/Italy/9097/1997 (H5N9) IAV (CI = 0.18 to 0.31). Taken together, the results suggest that regimens that combine neuraminidase inhibitors and nitazoxanide exert synergistic anti-influenza effects.
INTRODUCTION
Despite large immunization programs, influenza remains a public health concern, causing an average of 36,000 deaths annually in the United States and approximately 250,000 to 500,000 deaths worldwide (1). The etiological agent of the disease, the influenza viruses, are enveloped, negative-stranded RNA viruses, belonging to the family Orthomyxoviridae, classified in three types (A, B, and C), of which the A type is clinically the most important (2). The genomes of influenza A viruses consist of eight single-stranded RNA segments that encode a variety of structural and nonstructural proteins, including the main surface glycoproteins, hemagglutinin (HA) and neuraminidase (NA), of which 18 HA (H1 to H18) and 11 NA (NA1 to NA11) subtypes have been identified so far (2) and the transmembrane ion channel M2 protein, required for virus uncoating (3).
Influenza A viruses (IAV) represent a major health threat, since they may cross species barriers as a whole and adapt to a new host, having thus the potential to cause devastating pandemics (4). In March 2009, a novel strain of influenza A/H1N1 with the ability to transmit readily between humans emerged, resulting in a rapid global spread (5). In addition, the emergence of highly pathogenic avian influenza (HPAI) viruses in domestic poultry and the increasing number of cases of direct transmission of avian influenza viruses to humans represent a major risk, confirmed by the ongoing outbreak of HPAI H5N1 viruses in the bird population, which has caused a nearly 50% case fatality rate among the people infected (4–6). In addition to the H5N1 IAV, on 31 March 2013, China notified the WHO of the first human cases of a novel avian IAV (H7N9) causing severe illnesses, characterized by rapidly progressive pneumonia, respiratory failure, acute respiratory distress syndrome (ARDS), and fatal outcomes (7).
Whereas vaccines are the best option for the prophylaxis and control of a pandemic, the lag time between virus identification and vaccine distribution may exceed 6 months; concerns regarding vaccine safety are also a growing issue (8). In the short term, antiviral therapy is vital to control the spread of influenza.
Two therapies are currently licensed in the United States to treat influenza infections: the M2 ion channel blocker adamantane drugs (amantadine and rimantadine), which are now rarely used because of widespread resistance due to amino acid substitutions in the transmembrane domain of M2 protein (9), and the NA inhibitors oseltamivir (Tamiflu) and zanamivir (Relenza), which impair the efficient release of viruses from infected host cells. Also for NA inhibitors, in particular in the case of oseltamivir, drug-resistant variants continue to emerge (10). Because of the emergence of drug-resistant IAV strains, combination chemotherapy with drugs affecting different viral targets represents an attractive option for influenza treatment. On this basis, novel antiviral drugs effective against different strains of influenza viruses are therefore highly needed. In a context of multiple-drug therapy, it would be particularly valuable to develop drugs attacking the pathogen at a stage of the replication cycle different from that attacked by the currently available anti-influenza drugs.
We have previously shown that nitazoxanide (NTZ), a thiazolide anti-infective licensed in the United States (Alinia; Romark, Laboratories, USA) for treating enteritis caused by Cryptosporidium parvum and Giardia lamblia in children and adults (11–13), inhibits the replication of influenza A/Puerto Rico/8/1934 (H1N1) (PR8) virus in in vitro models (14). Importantly, thiazolides behave differently from other anti-influenza drugs, acting via a novel mechanism. These drugs do not affect virus infectivity, binding of or entry into target cells, and they do not cause a general inhibition of viral protein expression, whereas they selectively block the maturation and intracellular transport of the viral hemagglutinin (14). In particular, by using different biochemical approaches, we have shown that thiazolides block HA terminal glycosylation at a stage preceding resistance to endoglycosidase H digestion, which is a marker for transport into the cis and middle Golgi compartments. Immunomicroscopy studies and analysis of viral particles produced by infected cells also showed that the thiazolide-induced alterations impair HA0 trafficking between the endoplasmic reticulum and the Golgi complex, preventing its transport and insertion into the host cell plasma membrane, thus blocking the exit of mature virions from host cells (14).
In the present study, the antiviral activity of NTZ was tested against a variety of human and avian influenza A strains, including strains resistant to oseltamivir or amantadine, confirming that the drug is effective against all strains tested. Combination therapy studies were then undertaken to investigate whether NTZ could be combined additively, synergistically, or antagonistically with oseltamivir or zanamivir, using as a model the PR8 and A/WSN/1933 (H1N1) (WSN) IAVs and the avian low-pathogenicity strain A/chicken/Italy/9097/1997 (H5N9) (A/Ck) virus.
MATERIALS AND METHODS
Cell culture and treatments.
Madin-Darby canine kidney (MDCK) cells and human A549 alveolar type II-like epithelial cells (A549) (American Type Culture Collection, Manassas, VA) were grown at 37°C in a 5% CO2 atmosphere in RPMI 1640 medium (Gibco-Invitrogen, Carlsbad, CA) supplemented with 10% fetal calf serum (FCS), 2 mM glutamine, and antibiotics. Nitazoxanide (Romark Laboratories LC, Tampa, FL) dissolved in a dimethyl sulfoxide (DMSO) stock solution (25 mg/ml) was diluted in culture medium and added to infected cells immediately after a 1-h adsorption period. Controls received equal amounts of the DMSO vehicle (0.01 to 0.2% final concentration), which did not affect cell viability or virus replication. The NA inhibitors zanamivir, oseltamivir phosphate (oseltamivir) (Waterstone Technologies, Carmel, IN) and its active metabolite oseltamivir carboxylate (CHEMOS GmbH, Regenstauf, Germany) were dissolved in aqueous solution. For the combination studies, confluent cell monolayers were treated with different concentrations of the NA inhibitors for 30 min before infection, and treatment was repeated immediately after the virus adsorption period. All compounds were maintained in the medium for the duration of the experiment. Each concentration of each compound was tested in duplicate, and each experiment was repeated at least 3 times.
Cytotoxicity assay.
Cell viability was determined in quadruplicate in mock-infected cells treated with different concentrations of NTZ, oseltamivir phosphate, oseltamivir carboxylate, or zanamivir, alone or in combination, by the 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide (MTT) to MTT formazan conversion assay (Sigma-Aldrich), as previously described (14). All cytotoxicity assays were performed in mock-infected cells under the same culture conditions, including cell density and time of treatment, as the ones described for antiviral assays. The 50% lethal dose (LD50) was calculated using Prism 5.0 software (GraphPad Software Inc., San Diego, CA). Microscopic examination of mock-infected or virus-infected cells was performed using a Leica DM-IL microscope, and images were captured on a Leica DC 300 camera using Leica Image-Manager500 software.
Virus preparation and infection.
The following IAV strains were used in the study: A/Puerto Rico/8/1934 (H1N1) (PR8), A/WSN/1933 (H1N1) (WSN), amantadine-resistant A/Parma/06/2007 (H3N2) (AMD-R), oseltamivir-resistant A/Parma/24/2009 (H1N1) (OST-R), the human vaccine strain A/California/7/2009 (H1N1pdm09) (A/CA/7/09), and the avian low-pathogenicity A/chicken/Italy/9097/1997 (H5N9) (A/Ck), A/goose/Italy/296246/2003 (H1N1) (A/Gs), and A/turkey/Italy/RA5563/1999 (H7N1) (A/Tk) viruses. One influenza B virus, the B/Parma/3/2004 clinical isolate, was also tested. The resistant AMD-R and OST-R strains, the human A/CA/7/09 vaccine strain, the avian A/Ck, A/Gs, and A/Tk strains, and the B/Parma/3/2004 clinical isolate were a kind gift from Isabella Donatelli (Istituto Superiore di Sanità, Rome, Italy). Influenza A viruses were grown in the allantoic cavity of 8- or 10-day-old embryonated eggs (14, 15). After 48 h at 37°C, the allantoic fluid was harvested and centrifuged at 5,000 rpm for 30 min to remove cellular debris, and virus titers were determined by plaque assay according to standard procedures (16, 17). Confluent MDCK or A549 cell monolayers were mock infected or infected with the different influenza viruses for 1 h at 37°C at a multiplicity of infection (MOI) of 10 plaque forming units (PFU)/cell, unless differently specified. After the adsorption period, the viral inoculum was removed, and cell monolayers were washed three times with phosphate-buffered saline (PBS). Cells were maintained at 37°C in RPMI 1640 culture medium containing 2% FCS. For multistep virus growth curves, confluent MDCK or A549 cell monolayers were infected with the PR8 or WSN influenza virus for 1 h at 37°C at an MOI of 0.01 PFU/cell. After the 1-h adsorption period, the viral inoculum was removed, and cell monolayers were washed three times with PBS. Cells were maintained at 37°C in RPMI 1640 culture medium containing 0.5% bovine serum albumin (BSA) and l-1-tosylamide-2-phenylethyl chloromethyl ketone (TPCK)-treated trypsin (0.5 μg/ml) (Sigma-Aldrich).
Virus titration.
Virus yield was determined 24 and 48 h postinfection (p.i.) by HA titration as previously described (16), by plaque assay, or by a 50% tissue culture infective dose (TCID50) infectivity assay. For the plaque assay, serial 10-fold dilutions of the different IAVs were prepared and inoculated on confluent MDCK cell monolayers in 35-mm plates. After 1 h at 37°C, the inoculum was removed and the cells were washed three times with PBS before the addition of RPMI containing 0.5% BSA, 0.5 μg/ml TPCK-treated trypsin, and 0.6% SeaPlaque agarose (Lonza, Rockland, ME, USA). After 48 to 72 h (depending on the viral strain) at 37°C, the overlay was removed and cells were fixed with methanol and stained with 1% crystal violet (Sigma-Aldrich). For the TCID50 infectivity assay, MDCK cells grown on 96-well plates were inoculated with serial dilutions of viral suspension in the presence of 1 μg/ml TPCK-treated trypsin for 48 h at 37°C in a 5% CO2 atmosphere. The presence of virus replication was confirmed by immunocytochemical staining with anti-NP monoclonal antibodies (clone InA108, MAB1272; Abnova), as described previously (14). Prior to being stained, cells were fixed overnight in 2% phosphate-buffered formalin at 4°C and then washed 3 times with PBS. Subsequently, cells were incubated with 0.5% NP-40 (Sigma-Aldrich) for 20 min at 4°C. Following permeabilization, cells were washed 3 times with PBS and incubated overnight with a 1:250 dilution of anti-NP monoclonal antibodies at 4°C. After being washed with PBS, cells were incubated with a 1:300 dilution of an Alexa Fluor 488-conjugated secondary antibody (Invitrogen) for 1 h at room temperature. After staining, cells were examined using a Leica DM-IL fluorescence microscope equipped with UV excitation filters. Virus titers, expressed as TCID50s/ml, were calculated by the method of Reed and Muench, as described previously (17). The 50% inhibitory concentrations (IC50s) and IC90s of the different antiviral compounds were calculated using Prism 5.0 software.
Statistical analysis and drug synergism studies.
Statistical analysis was performed using the Student t test for unpaired data. Data are expressed as the means ± standard deviations (SD) of results from quadruplicate samples. P values of <0.05 were considered significant.
For the analysis of drug synergism, the effect of treatment of MDCK and A549 cells with NTZ and the NA inhibitors, alone or in combination, on influenza A virus infection was investigated using isobologram analysis, according to the Chou-Talalay theory that provides algorithms for automated computer simulation for synergism and/or antagonism based on the median-effect equation derived from the mass action law (18, 19) and by CalcuSyn Windows software for dose-effect analysis and synergism/antagonism quantification (Biosoft, Cambridge, United Kingdom) (20). The conservative isobologram assumes that two drugs have independent or dissimilar modes of actions (21). For the constant-ratio (IC50 ratio) two-drug combination design, the combination of the drugs at their equipotent ratio (the ratio of their IC50s) was chosen. After the ratio was set, a mixture of the two drugs at 4-fold their IC50s was prepared. The mixture was serially diluted (2-fold dilutions, 4:2, 2:1, 1:0.5, and 0.5:0.25) to obtain a wide dosage range, as indicated in the CalcuSyn standard protocol. For single-drug treatment, serial dilutions were prepared from the 4-fold-concentrated stock of each compound, using the same protocol.
Dose inhibition curves were drawn for NTZ, oseltamivir, oseltamivir carboxylate, and zanamivir, used alone or in combination. For each drug combination, the IC50 values were plotted against the fractional concentrations of the NA inhibitor and NTZ on the x axis and y axis, respectively. IC50, IC75, and IC90 values were determined using CalcuSyn software, which performs single- and multiple-drug dose-effect calculations and determines the presence of antagonism, additivity, or synergism. Using the median-effect equation, this program was used to plot dose-effect curves for each drug and combination of drugs and to calculate the combination index (CI). For two-drug combinations, in the isobologram plot, if the combination data point (indicated with a symbol in Fig. 4 to 6) falls on the diagonal, an additive effect is indicated; if it falls on the lower left, synergism is indicated, whereas if it falls on the upper right, antagonism is indicated. In addition, a CI of >1 denotes antagonism, a CI equal to 1 denotes additivity, and a CI of <1 denotes synergism. In particular, CI values in the range of 0.1 to 0.3, 0.3 to 0.7, and 0.7 to 0.85 are considered to indicate strong synergism, synergism, and moderate synergism, respectively. The evaluation of drug synergism based on a median-effect equation has been widely employed in the literature.
RESULTS
Anti-influenza activity of nitazoxanide.
The effect of treatment with nitazoxanide (Fig. 1a) was initially investigated in MDCK cells after infection with the A/Puerto Rico/8/1934 (H1N1) (PR8) virus. Confluent monolayers of MDCK cells mock infected or infected with PR8 virus (10 PFU/cell) were treated with different concentrations of NTZ (0.32, 3.2, 32, and 160 μM) or vehicle immediately after the virus adsorption period, and virus yield was determined at 24 h p.i. by HA titration and, in parallel, by plaque assay and TCID50 infectivity assay. At the same time, cell viability was determined by MTT assay of mock-infected cells. NTZ treatment caused a dose-dependent inhibition of virus replication, and similar IC50s in the low (1.3 to 4.1) μM range were obtained by HA titration, TCID50 assay, and plaque assay (Table 1). HA titration was thus used for the following studies, unless differently specified.
TABLE 1.
PR8 virus yield | IC50 (μM) | LD50 (μM) | SI |
---|---|---|---|
HAU/ml | 3.2 ± 0.0 | >163.0 | >50.9 |
TCID50/ml | 4.1 ± 0.2 | >163.0 | >39.8 |
PFU/ml | 1.3 ± 0.1 | >163.0 | >125.4 |
PR8 IAV yield was determined at 24 h p.i. by HA titration, TCID50 infectivity assay, and plaque assay. Data are the means ± SD of results from quadruplicate samples.
As previously reported (14), the antiviral activity was independent of the MOI used in MDCK cells (Fig. 1b and c). To investigate whether NTZ treatment was equally effective in human cells, confluent monolayers of alveolar type II-like epithelial A549 cells were infected with PR8 virus under single-step (10 PFU/cell) and multistep (0.01 PFU/cell) growth conditions and were treated with different concentrations of NTZ or vehicle immediately after the virus adsorption period. Virus yield was determined at 24 h or 48 h p.i. by HA titration. As shown in Fig. 1b and c, NTZ was effective in inhibiting IAV replication also in A549 cells under both single-step and multistep growth conditions.
In addition to PR8 virus, the antiviral activity of NTZ was investigated in MDCK cells against different influenza A strains, including A/WSN/1933 (H1N1), amantadine-resistant A/Parma/06/2007 (H3N2), oseltamivir-resistant A/Parma/24/2009 (H1N1), the human vaccine strain A/California/7/2009 (H1N1pdm09), and the avian low-pathogenicity A/chicken/Italy/9097/1997 (H5N9), A/goose/Italy/296246/2003 (H1N1), and A/turkey/Italy/RA5563/1999 (H7N1) strains. Soon after the adsorption period, cells were treated with 30 μM NTZ, and virus yield was determined 24 h p.i. by plaque assay. At the same time, the viability of mock-infected cells was determined by the MTT assay. The results, shown in Fig. 2, indicate that NTZ treatment inhibited the replication of all IAV strains tested, including the amantadine-resistant and the oseltamivir-resistant strains, at noncytotoxic doses (LD50s, >163 μM in all cases). Similar results were obtained in parallel samples after determination of virus yield by HA titration (data not shown).
In a different experiment, confluent monolayers of MDCK cells were infected with 5 hemagglutination units (HAU)/105 cells of the different mammalian and avian IAV strains for 1 h or mock infected. After this time, cells were treated with different concentrations (0.32, 3.2, 32, and 160 μM) of NTZ. At 24 h p.i., virus yield was determined by HA titration, and the cell viability was determined by the MTT assay in mock-infected cells. As shown in Table 2, NTZ was effective against all IAV strains tested, with IC50s ranging from 1 to 3.2 μM and selectivity indexes (SIs) ranging from >50.9 to >163.
TABLE 2.
Virus | IC50 (μM) | IC90 (μM) | LD50 (μM) | SI |
---|---|---|---|---|
PR8 | 3.2 ± 0.0 | 26.0 ± 2.5 | >163.0 | >50.9 |
WSN | 1.6 ± 0.2 | 16.3 ± 2.3 | >163.0 | >101.9 |
CA09 | 3.2 ± 0.0 | 20.8 ± 0.0 | >163.0 | >50.9 |
OST-R | 1.9 ± 0.0 | 21.1 ± 0.0 | >163.0 | >85.8 |
AMD-R | 1.0 ± 0.0 | 13.0 ± 0.0 | >163.0 | >163.0 |
A/Ck | 3.2 ± 0.5 | 26.0 ± 2.3 | >163.0 | >50.9 |
A/Gs | 3.2 ± 0.2 | 26.8 ± 2.6 | >163.0 | >50.9 |
A/Tk | 1.6 ± 0.2 | 20.5 ± 4.7 | >163.0 | >101.9 |
Virus yield was determined at 24 h p.i. by HA titration. Data are the means ± SD of results from quadruplicate samples. PR8, A/Puerto Rico/8/1934 (H1N1); WSN, A/WSN/1933 (H1N1); CA09, A/California/7/2009 (H1N1pdm09); OST-R, oseltamivir-resistant A/Parma/24/2009 (H1N1); AMD-R, amantadine-resistant A/Parma/06/2007 (H3N2); A/Ck, A/chicken/Italy/9097/1997 (H5N9); A/Gs, A/goose/Italy/296246/2003 (H1N1); A/Tk, A/turkey/Italy/RA5563/1999 (H7N1).
Interestingly, NTZ was equally effective against the B/Parma/3/2004 clinical isolate (IC50 = 1 μM, SI > 163), indicating good antiviral activity also against influenza B viruses.
Effect of the nitazoxanide and oseltamivir combination treatment in influenza A virus-infected cells.
The fact that NTZ was shown to act at a different level than NA inhibitors (14) and was effective against the oseltamivir-resistant A/Parma/24/2009 (H1N1) strain prompted us to investigate a possible synergistic antiviral effect between the thiazolide and oseltamivir.
In a first set of experiments, confluent MDCK cells infected with PR8 virus (10 PFU/cell) were treated with NTZ (10 μM) and oseltamivir (10 μM) alone or in combination immediately after the virus adsorption period, and virus yield was determined at 24 h p.i. by plaque assay. As shown in Fig. 3, the NTZ-oseltamivir combination treatment was significantly more effective than monotherapy. To confirm a synergistic antiviral effect between the thiazolide and oseltamivir, confluent MDCK cells were mock infected or infected with the PR8 virus at an MOI of 10 PFU/cell and the effect of NTZ and oseltamivir used alone or in combination was first investigated under single-step virus growth conditions with a constant-ratio (IC50 ratio) two-drug combination design using isobologram analysis according to the Chou-Talalay method, as described in Materials and Methods. Virus yield was determined 24 h p.i. by HA titration. A log dose-effect curve and median-effect plot were generated for both drugs using the CalcuSyn software (Fig. 4a). The x intercept of the median-effect plot determines the median-effect dose (Dm), which, under the conditions described, is equivalent to the IC50; the Dm were found to be 4.02 μM and 4.64 μM for NTZ and oseltamivir, respectively, when administered as monotherapy. The Dm of the two-drug combination in a molar ratio of 3.2 to 10 resulted to be 0.57 μM and 1.8 μM for NTZ and oseltamivir, respectively, indicating that, when the drugs are used in combination, the doses needed to achieve 50% viral growth inhibition were approximately 7-fold and 2.6-fold lower than those needed in monotherapy for NTZ and oseltamivir, respectively.
CalcuSyn-generated isobolograms based on the two drugs administered in combination at fixed ratios are shown in Fig. 4b. The ED50, ED75, and ED90 (50%, 75%, and 90% equipotent doses) plots for each drug ratio fell below the line representing additivity, indicating a synergistic effect of the drug combination on IAV replication. CI values were calculated for all conditions. The NTZ-oseltamivir combination treatment was synergistic, with CI values ranging between 0.39 and 0.53 (Table 3). As determined by the MTT assay, there was no significant difference in the viabilities of mock-infected cells at the different drug concentrations tested (data not shown), indicating that the synergistic action of NTZ and oseltamivir was due to their pharmacological effect and not to increased cytotoxicity.
TABLE 3.
Drug | Expt | MOI (PFU) | CI at indicated ED in: |
|||||
---|---|---|---|---|---|---|---|---|
PR8-MDCK cells |
WSN-A549 cells |
|||||||
ED50 | ED75 | ED90 | ED50 | ED75 | ED90 | |||
NTZ-OST | Single step | 10 | 0.53 | 0.44 | 0.39 | 0.47 | 0.51 | 0.56 |
Multistep | 0.01 | 0.47 | 0.54 | 0.63 | 0.51 | 0.56 | 0.62 | |
NTZ-OST-C | Single step | 10 | 0.73 | 0.72 | 0.71 | 0.27 | 0.45 | 0.75 |
Multistep | 0.01 | 0.44 | 0.47 | 0.51 | 0.41 | 0.40 | 0.39 |
Virus yield was determined by HA titration at 24 h p.i. for single-step and 48 h p.i. for multistep experiments. Combination index (CI) values were generated by CalcuSyn software from duplicate samples from a representative experiment of three with similar results. NTZ-OST, nitazoxanide-oseltamivir phosphate; NTZ-OST-C, nitazoxanide-oseltamivir carboxylate.
The effects of NTZ, oseltamivir, and combinations of both drugs were also investigated under multistep virus growth conditions, as described in Materials and Methods. Virus yield was determined 24 and 48 h p.i. Median-effect curves and conservative isobolograms (Fig. 4c and d) were plotted with the CalcuSyn software as described for single-step virus growth experiments. Both NTZ and oseltamivir were more effective under multistep conditions than under single-step growth conditions, with IC50s of 0.9 and 4.1 μM, respectively, when administered as single drugs. Isobologram analysis revealed that the combination of the two drugs was synergistic, with CI values in the range of 0.47 to 0.63 (Table 3). Similar results were obtained at 48 h p.i., with CI values ranging from 0.46 to 0.66 (data not shown). A synergistic antiviral effect was obtained in a parallel set of experiments using the active metabolite of oseltamivir, oseltamivir carboxylate. Isobologram analysis revealed that the combination of the two drugs was synergistic under both single-step and multistep virus growth conditions, with CI values in the range of 0.44 to 0.73 (Table 3).
To investigate whether the antiviral activity was dependent on the cell type and the viral strain, confluent A549 cells were mock infected or infected with WSN virus under single-step and multistep virus growth conditions, and the effect of NTZ and oseltamivir, as well as NTZ and oseltamivir carboxylate, used alone or in combination, was investigated in a constant-ratio two-drug combination design according to the Chou-Talalay method, as described above. As shown in Table 3, the combined treatment was synergistic under both single-step and multistep virus growth conditions, with CI values in the range of 0.27 to 0.75. As determined by the MTT assay, also in this model, there was no significant difference in the levels of viability of mock-infected cells at the different drug concentrations tested (data not shown).
Synergistic anti-IAV activity of the nitazoxanide and zanamivir combination.
To assess whether a similar synergistic antiviral effect could be obtained with the anti-influenza drug zanamivir, the effects of NTZ, zanamivir, and a combination of both drugs were investigated under single-step virus growth conditions. Confluent MDCK cell monolayers were mock infected or infected with PR8 virus (10 PFU/cell) and treated as described in Materials and Methods. Compounds were maintained in the medium for the duration of the experiment, and virus yield was determined 24 h p.i. Log dose-effect curves and median-effect plots were generated for both drugs using the CalcuSyn software, as described above (Fig. 5a and c). Median-effect curves and conservative isobologram analysis indicated that the combination treatment was synergistic, with CI values in the 0.31-to-0.48 range (Fig. 5b and d). As determined by the MTT assay, also in this case, there was no significant difference in the levels of viability of mock-infected cells at the different drug concentrations tested (data not shown), indicating that the synergistic action of NTZ and zanamivir was not due to increased cytotoxicity.
Analysis of nitazoxanide and oseltamivir synergism in cells infected with the avian A/Ck IAV.
To investigate whether the combination of NTZ and a neuraminidase inhibitor exerts synergistic, additive, or antagonistic antiviral effects against avian influenza A viruses, MDCK cells were mock infected or infected with the avian H5N9 low-pathogenicity strain A/chicken/Italy/9097/1997 (H5N9) and treated with NTZ and oseltamivir as described in Materials and Methods. The antiviral activities of NTZ and oseltamivir alone and in combination were analyzed under single-step virus growth conditions at an MOI of 2 PFU/cell. As shown in Fig. 6, oseltamivir was found to be more effective against the A/Ck IAV strain than against the PR8 strain, with a Dm of 0.33 μM when administered as a single drug. Under these conditions, the NTZ Dm was 2.74 μM when it was administered as monotherapy, whereas the Dm of the two-drug combination in a molar ratio 16:1 was found to be 0.33 μM and 0.02 μM for NTZ and oseltamivir, respectively, indicating that to achieve a 50% inhibition of virus yield, much lower doses of NTZ and oseltamivir were needed (Fig. 6a and c). Also in this model, the NTZ-oseltamivir combination treatment was synergistic, as indicated by the isobologram analysis and the CI values, which were found to be in the 0.18-to-0.31 range (Fig. 6b and d).
DISCUSSION
The emergence of highly pathogenic influenza A virus strains represents a serious threat to global human health. In the case of the avian H7N9 influenza virus recently isolated in China, as of 27 June 2014, 450 total laboratory-confirmed cases have been reported, resulting in 165 fatalities (22).
Efforts to control emerging influenza strains are focused on surveillance and early diagnosis, as well as on the development of effective vaccines and novel antiviral drugs. Of the two classes of drugs currently available for chemoprophylaxis and treatment of influenza, the neuraminidase inhibitors oseltamivir and zanamivir, as well as the novel sialic acid analogs, have prophylactic effect but moderate therapeutic effects, whereas the M2 ion channel inhibitors amantadine and rimantadine reduce the duration of symptoms of clinical influenza but may present major side effects (23, 24). An additional obstacle to effective treatment of influenza with these drugs is represented by the emergence of resistance to both types of inhibitors. In the case of adamantanes, for example, the Centers for Disease Control and Prevention (CDC) in the United States reported that 100% of seasonal H3N2 and 99.8% of the 2009 pandemic influenza samples tested for the 2008-2009 influenza season showed resistance to these drugs (25). In the case of NA inhibitors, in 2008 to 2009 in most parts of the world, seasonal human influenza A/H1N1 virus strains were found to carry the H274Y amino acid substitution in the neuraminidase gene, which causes resistance to oseltamivir (26, 27). An amino acid change in the viral neuraminidase (NA-R292K) (N2 numbering) associated with high resistance to oseltamivir and peramivir and partial resistance to zanamivir was recently found also in some H7N9 clinical isolates (28, 29). It was also shown that the acquisition of high-level oseltamivir resistance due to the NA-R292K substitution does not substantially alter H7N9 IAV virulence or transmissibility (29). Intriguingly, it was also demonstrated that oseltamivir carboxylate, the excreted metabolite of oseltamivir that can be found in sewage, may be able to confer drug resistance to viral populations that have infected wild ducks that live in close contact with water in affected environments (30).
For the reasons discussed above, the disease is by no means under control, and novel antiviral drugs designed against different influenza virus targets are urgently needed.
As indicated in the introduction, nitazoxanide is a safe, orally bioavailable anti-infective drug licensed in the United States for treating infections by Cryptosporidium parvum and Giardia lamblia in children and adults (31). In addition to treating protozoan and bacterial infections, thiazolides have emerged as a new class of broad-spectrum antiviral drugs (14, 32–36). We have previously reported that NTZ and its active circulating metabolite tizoxanide possess anti-influenza activity and have shown that the thiazolides inhibit H1N1 IAV replication by a novel mechanism (14). As indicated in the introduction, these molecules act in fact at the posttranslational level by selectively blocking the maturation of the viral hemagglutinin at a stage preceding resistance to endoglycosidase H digestion and impairing HA intracellular trafficking and insertion into the host cell plasma membrane, a key step for correct assembly and exit of the virus from the host cell (14). Recently, NTZ was found to be effective against influenza A (H3N2) variant [A (H3N2)v] viruses that share the A (H1N1)pdm09 M gene, containing the marker of M2 blocker resistance, S31N (37).
We now report that nitazoxanide is effective against a variety of human and avian influenza A virus strains, including an H7N1 avian IAV strain. We also show that the drug is effective against an amantadine-resistant IAV strain, A/Parma/06/2007 (H3N2), and an oseltamivir-resistant strain, A/Parma/24/2009 (H1N1). This last finding prompted us to investigate whether nitazoxanide could exert synergistic antiviral effect against IAV when given in combination with a neuraminidase inhibitor.
In fact, combination chemotherapy with two or more drugs that target different influenza proteins or host factors may represent an attractive approach for counteracting drug resistance and overcoming the limitations of monotherapy. In addition, multiple-drug therapy may enhance clinical outcomes by allowing a reduction of individual drug doses, thus decreasing dose-related drug toxicity (38).
Combination therapy studies were therefore undertaken to investigate the effect of NTZ when administered in combination with oseltamivir or zanamivir, using as a model the H1N1 PR8 IAV strain in MDCK cells; the results were analyzed using isobologram analysis according to the Chou-Talalay method and CalcuSyn software for dose-effect analysis and synergism/antagonism quantification (18–21). The NTZ-oseltamivir combination treatment resulted in synergism, with CI values ranging between 0.39 and 0.63, independently of the MOI used. There was no significant difference in the levels of viability of mock-infected cells at the different drug concentrations tested, indicating that the synergistic action of NTZ and oseltamivir could not be ascribed to increased cytotoxicity. In addition, a synergistic antiviral effect was achieved using NTZ in combination with oseltamivir carboxylate in the same model and in human A549 cells infected with A/WSN/1933 (H1N1) IAV under single-step and multistep virus growth conditions.
A synergistic antiviral effect was obtained also when NTZ was administered in combination with zanamivir. Median-effect curves and conservative isobologram analysis indicated that the NTZ-zanamivir combination treatment was synergistic, with CI values in the 0.3-to-0.48 range.
Finally, to investigate whether the combination of NTZ and NA inhibitors could exert a synergistic antiviral effect also against avian influenza A viruses, a similar set of experiments was performed using the avian low-pathogenicity H5N9 A/Ck IAV strain in MDCK cells. Also in this experimental model, the NTZ-oseltamivir combination treatment was synergistic with CI values in the 0.18-to-0.31 range.
In addition to in vitro studies (14, 37), a randomized double-blind, placebo-controlled phase 2B study in adults and adolescents with acute uncomplicated influenza recently showed the anti-influenza activity of nitazoxanide (39). This study also indicated that concentrations of the nitazoxanide metabolite tizoxanide of 2 to 60 μg/ml, efficacious for IAV infections in vitro, are achieved in the plasma of patients treated with 300 to 600 mg nitazoxanide twice daily for 5 days (39).
It should be pointed out that the results described herein were the basis for designing the clinical study protocol of NTZ in the treatment of uncomplicated influenza A and B, which is in late phase 3 clinical trial in the United States, Canada, Australia, and New Zealand, supported by the U.S. Department of Health and Human Services and managed by the Biomedical Advanced Research & Development Authority (BARDA). The study (the ClinicalTrials.gov identifier is NCT01610245) is a double-blind, placebo-controlled study of 2,000 adults and adolescents from 12 to 65 years of age, one of the largest phase 3 clinical trials ever conducted for the treatment of acute uncomplicated influenza, comprising a combination of anti-influenza drugs.
The new finding described herein that nitazoxanide exerts synergistic antiviral activity against human and avian influenza viruses when given in combination with oseltamivir or zanamivir reinforces the concept that thiazolides may provide an important new addition to the arsenal of drugs used for chemoprophylaxis and treatment of influenza.
ACKNOWLEDGMENTS
We thank Isabella Donatelli (Istituto Superiore di Sanità, Rome, Italy) for kindly providing the B/Parma/3/2004 clinical isolate and the following IAV strains: A/Parma/06/2007 (H3N2), A/Parma/24/2009 (H1N1), A/California/7/2009 (H1N1pdm09), A/chicken/Italy/9097/1997 (H5N9), A/goose/Italy/296246/2003 (H1N1), and A/turkey/Italy/RA5563/1999 (H7N1).
This work was financially supported by Romark Laboratories LC, Tampa, FL, and by a grant from the Italian Ministry of University and Scientific Research (PRIN project N 2010PHT9NF-006). Romark Laboratories LC is the company that owns the intellectual property rights related to nitazoxanide. J. F. Rossignol is an employee and stockholder of Romark Laboratories, LC; all authors have served as consultants to Romark Laboratories, LC.
REFERENCES
- 1.Rappuoli L, Dormitzer PR. 2012. Influenza: options to improve pandemic preparation. Science 336:1631–1633. doi: 10.1126/science.336.6089.1631. [DOI] [PubMed] [Google Scholar]
- 2.Tong S, Zhu X, Li Y, Shi M, Zhang J, Bourgeois M, Yang H, Chen X, Recuenco S, Gomez J, Chen LM, Johnson A, Tao Y, Dreyfus C, Yu W, McBride R, Carney PJ, Gilbert AT, Chang J, Guo Z, Davis CT, Paulson JC, Stevens J, Rupprecht CE, Holmes EC, Wilson IA, Donis RO. 2013. New world bats harbor diverse influenza A viruses. PLoS Pathog 9:e1003657. doi: 10.1371/journal.ppat.1003657. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Shaw ML, Palese P. 2013. Orthomyxoviridae, p 1151–1243. In Knipe DM, Howley PM (ed), Fields virology, 6th ed, vol 1 Lippincott Williams & Wilkins, Philadelphia, PA. [Google Scholar]
- 4.Horimoto T, Kawaoka Y. 2005. Influenza: lessons from past pandemics, warnings from current incidents. Nat Rev Microbiol 3:591–600. doi: 10.1038/nrmicro1208. [DOI] [PubMed] [Google Scholar]
- 5.Fraser C, Donnelly CA, Cauchemez S, Hanage WP, Van Kerkhove MD, Hollingsworth TD, Griffin J, Baggaley RF, Jenkins HE, Lyons EJ, Jombart T, Hinsley WR, Grassly NC, Balloux F, Ghani AC, Ferguson NM, Rambaut A, Pybus OG, Lopez-Gatell H, Alpuche-Aranda CM, Chapela IB, Zavala EP, Guevara DM, Checchi F, Garcia E, Hugonnet S, Roth C, Rapid Pandemic Assessment Collaboration WHO . 2009. Pandemic potential of a strain of influenza A (H1N1): early findings. Science 324:1557–1561. doi: 10.1126/science.1176062. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.de Jong MD, Hien TT. 2006. Avian influenza A (H5N1). J Clin Virol 35:2–13. doi: 10.1016/j.jcv.2005.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Gao R, Cao B, Hu Y, Feng Z, Wang D, Hu W, Chen J, Jie Z, Qiu H, Xu K, Xu X, Lu H, Zhu W, Gao Z, Xiang N, Shen Y, He Z, Gu Y, Zhang Z, Yang Y, Zhao X, Zhou L, Li X, Zou S, Zhang Y, Li X, Yang L, Guo J, Dong J, Li Q, Dong L, Zhu Y, Bai T, Wang S, Hao P, Yang W, Zhang Y, Han J, Yu H, Li D, Gao GF, Wu G, Wang Y, Yuan Z, Shu Y. 2013. Human infection with a novel avian-origin influenza A (H7N9) virus. N Engl J Med 368:1888–1897. doi: 10.1056/NEJMoa1304459. [DOI] [PubMed] [Google Scholar]
- 8.Boltz DA, Aldridge JR Jr, Webster RG, Govorkova EA. 2010. Drugs in development for influenza. Drugs 70:1349–1362. doi: 10.2165/11537960-000000000-00000. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Pielak RM, Schnell JR, Chou JJ. 2009. Mechanism of drug inhibition and drug resistance of influenza A M2 channel. Proc Natl Acad Sci U S A 106:7379–7384. doi: 10.1073/pnas.0902548106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Thorlund K, Awad T, Boivin G, Thabane L. 2011. Systematic review of influenza resistance to the neuraminidase inhibitors. BMC Infect Dis 11:134. doi: 10.1186/1471-2334-11-134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Fox LM, Saravolatz LD. 2005. Nitazoxanide: a new thiazolide antiparasitic agent. Clin Infect Dis 40:1173–1180. doi: 10.1086/428839. [DOI] [PubMed] [Google Scholar]
- 12.Rossignol JF, Kabil SM, el-Gohary Y, Younis AM. 2006. Effect of nitazoxanide in diarrhea and enteritis caused by Cryptosporidium species. Clin Gastroenterol Hepatol 4:320–324. doi: 10.1016/j.cgh.2005.12.020. [DOI] [PubMed] [Google Scholar]
- 13.Rossignol JF, Ayoub A, Ayers MS. 2001. Treatment of diarrhea caused by Giardia intestinalis and Entamoeba histolytica or E. dispar: a randomized, double-blind, placebo-controlled study of nitazoxanide. J Infect Dis 184:381–384. doi: 10.1086/322038. [DOI] [PubMed] [Google Scholar]
- 14.Rossignol JF, La Frazia S, Chiappa L, Ciucci A, Santoro MG. 2009. Thiazolides, a new class of anti-influenza molecules targeting viral hemagglutinin at the post-translational level. J Biol Chem 284:29798–29808. doi: 10.1074/jbc.M109.029470. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Donatelli I, Campitelli L, Di Trani L, Puzelli S, Selli L, Fioretti A, Alexander DJ, Tollis M, Krauss S, Webster RG. 2001. Characterization of H5N2 influenza viruses from Italian poultry. J Gen Virol 82:623–630. doi: 10.1099/vir.0.17377-0. [DOI] [PubMed] [Google Scholar]
- 16.Bernasconi D, Amici C, La Frazia S, Ianaro A, Santoro MG. 2005. The IκB kinase is a key factor in triggering influenza A virus-induced inflammatory cytokine production in airway epithelial cells. J Biol Chem 280:24127–24134. doi: 10.1074/jbc.M413726200. [DOI] [PubMed] [Google Scholar]
- 17.Pica F, Palamara AT, Rossi A, De Marco A, Amici C, Santoro MG. 2000. Delta(12)-prostaglandin J(2) is a potent inhibitor of influenza A virus replication. Antimicrob Agents Chemother 44:200–204. doi: 10.1128/AAC.44.1.200-204.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Chou TC, Talalay P. 1983. Analysis of combined drug effects: a new look at a very old problem. Trends Pharmacol Sci 4:450–454. doi: 10.1016/0165-6147(83)90490-X. [DOI] [Google Scholar]
- 19.Chou TC. 2010. Drug combination studies and their synergy quantification using the Chou-Talalay method. Cancer Res 70:440–446. doi: 10.1158/0008-5472.CAN-09-1947. [DOI] [PubMed] [Google Scholar]
- 20.Chou TC, Hayball M, Lamble CW. 1996. CalcuSyn—Windows software for dose-effect analysis and synergism/antagonism quantification, and user's manual. Biosoft, Cambridge, United Kingdom. [Google Scholar]
- 21.Chou TC, Talalay P. 1981. Generalized equations for the analysis of inhibitors of Michaelis-Menten and higher order kinetic systems with two or more mutually exclusive and nonexclusive inhibitors. Eur J Biochem 115:207–216. [DOI] [PubMed] [Google Scholar]
- 22.World Health Organization (WHO). 2014. Human infections with avian influenza A (H7N9) virus. WHO risk assessment 27 June 2014. http://www.who.int/influenza/human_animal_interface/influenza_h7n9/riskassessment_h7n9_27june14.pdf.
- 23.von Itzstein M. 2007. The war against influenza: discovery and development of sialidase inhibitors. Nat Rev Drug Discov 6:967–974. doi: 10.1038/nrd2400. [DOI] [PubMed] [Google Scholar]
- 24.Regoes RR, Bonhoeffer S. 2006. Emergence of drug-resistant influenza virus: population dynamical considerations. Science 312:389–391. doi: 10.1126/science.1122947. [DOI] [PubMed] [Google Scholar]
- 25.Centers for Disease Control and Prevention. 2010. 2009-2010 influenza season summary. Centers for Disease Control and Prevention, Atlanta, GA; http://www.cdc.gov/flu/weekly/weeklyarchives2009-2010/09-10Summary.htm. [Google Scholar]
- 26.Meijer A, Lackenby A, Hungnes O, Lina B, van der Werf S, Schweiger B, Opp M, Paget J, van de Kassteele J, Hay A, Zambon M, European Influenza Surveillance Scheme . 2009. Oseltamivir-resistant influenza virus A (H1N1), Europe, 2007–08 season. Emerg Infect Dis 15:552–560. doi: 10.3201/eid1504.081280. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Moscona A. 2009. Global transmission of oseltamivir-resistant influenza. N Engl J Med 360:953–956. doi: 10.1056/NEJMp0900648. [DOI] [PubMed] [Google Scholar]
- 28.Hu Y, Lu S, Song Z, Wang W, Hao P, Li J, Zhang X, Yen HL, Shi B, Li T, Guan W, Xu L, Liu Y, Wang S, Zhang X, Tian D, Zhu Z, He J, Huang K, Chen H, Zheng L, Li X, Ping J, Kang B, Xi X, Zha L, Li Y, Zhang Z, Peiris M, Yuan Z. 2013. Association between adverse clinical outcome in human disease caused by novel influenza A H7N9 virus and sustained viral shedding and emergence of antiviral resistance. Lancet 381:2273–2279. doi: 10.1016/S0140-6736(13)61125-3. [DOI] [PubMed] [Google Scholar]
- 29.Hai R, Schmolke M, Leyva-Grado VH, Thangavel RR, Margine I, Jaffe EL, Krammer F, Solórzano A, García-Sastre A, Palese P, Bouvier NM. 2013. Influenza A(H7N9) virus gains neuraminidase inhibitor resistance without loss of in vivo virulence or transmissibility. Nat Commun 4:2854. doi: 10.1038/ncomms3854. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Järhult JD, Muradrasoli S, Wahlgren J, Söderström H, Orozovic G, Gunnarsson G, Bröjer C, Latorre-Margalef N, Fick J, Grabic R, Lennerstrand J, Waldenström J, Lundkvist A, Olsen B. 2011. Environmental levels of the antiviral oseltamivir induce development of resistance mutation H274Y in influenza A/H1N1 virus in mallards. PLoS One 6:e24742. doi: 10.1371/journal.pone.0024742. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.White CA., Jr 2004. Nitazoxanide: a new broad spectrum antiparasitic agent. Expert Rev Anti Infect Ther 2:43–49. doi: 10.1586/14787210.2.1.43. [DOI] [PubMed] [Google Scholar]
- 32.Rossignol JF, Abu-Zekry M, Hussein A, Santoro MG. 2006. Effect of nitazoxanide for treatment of severe rotavirus diarrhoea: randomised double-blind placebo-controlled trial. Lancet 368:124–129. doi: 10.1016/S0140-6736(06)68852-1. [DOI] [PubMed] [Google Scholar]
- 33.La Frazia S, Ciucci A, Arnoldi F, Coira M, Gianferretti P, Angelini M, Belardo G, Burrone OR, Rossignol JF, Santoro MG. 2013. Thiazolides, a new class of antiviral agents effective against rotavirus infection, target viral morphogenesis, inhibiting viroplasm formation. J Virol 87:11096–11106. doi: 10.1128/JVI.01213-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Korba BE, Montero AB, Farrar K, Gaye K, Mukerjee S, Ayers MS, Rossignol JF. 2008. Nitazoxanide, tizoxanide and other thiazolides are potent inhibitors of hepatitis B virus and hepatitis C virus replication. Antiviral Res 77:56–63. doi: 10.1016/j.antiviral.2007.08.005. [DOI] [PubMed] [Google Scholar]
- 35.Rossignol JF, Elfert A, El-Gohary Y, Keeffe EB. 2009. Improved virologic response in chronic hepatitis C genotype 4 treated with nitazoxanide, peginterferon, and ribavirin. Gastroenterology 136:856–862. doi: 10.1053/j.gastro.2008.11.037. [DOI] [PubMed] [Google Scholar]
- 36.Elazar M, Liu M, McKenna SA, Liu P, Gehrig EA, Puglisi JD, Rossignol JF, Glenn JS. 2009. The anti-hepatitis C agent nitazoxanide induces phosphorylation of eukaryotic initiation factor 2alpha via protein kinase activated by double-stranded RNA activation. Gastroenterology 137:1827–1835. doi: 10.1053/j.gastro.2009.07.056. [DOI] [PubMed] [Google Scholar]
- 37.Sleeman K, Mishin VP, Guo Z, Garten RJ, Balish A, Fry AM, Villanueva J, Stevens J, Gubareva LV. 2014. Antiviral susceptibility of variant influenza A (H3N2)v viruses isolated in the United States from 2011 to 2013. Antimicrob Agents Chemother 58:2045–2051. doi: 10.1128/AAC.02556-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Govorkova EA, Webster RG. 2010. Combination chemotherapy for influenza. Viruses 2:1510–1529. doi: 10.3390/v2081510. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Haffizulla J, Hartman A, Hoppers M, Resnick H, Samudrala S, Ginocchio C, Bardin M, Rossignol JF, Nitazoxanide Influenza Clinical Study Group US . 2014. Effect of nitazoxanide in adults and adolescents with acute uncomplicated influenza: a double-blind, randomised, placebo-controlled, phase 2b/3 trial. Lancet Infect Dis 14:609–618. doi: 10.1016/S1473-3099(14)70717-0. [DOI] [PMC free article] [PubMed] [Google Scholar]